NewStats: 3,265,530 , 8,187,099 topics. Date: Sunday, 15 June 2025 at 03:25 PM 596n6o

6382y

Backend Software Engineers, Let's Talk About System Design - Programming - Nairaland 326q5p

Backend Software Engineers, Let's Talk About System Design (11966 Views)

(4)

(1) Go Down)

SilverG33k(m): 9:26pm On Dec 05, 2020
Honestly, let's just catch cruise here.

Now here's a quick question, you're to build nairaland backend, how would you do that?

What technologies would you use, how would you handle scalability issues. Note than, the expected traffic for this system is 1k req/s just for fun tho

No yabbing

8 Likes 3 Shares

talk2hb1(m): 12:50am On Dec 06, 2020
Tooh,
Waiting for Experience Bosses to Come Tutor us on this.

3 Likes 1 Share

ULOPVQ: 4:51am On Dec 06, 2020
DC49: 6:40am On Dec 06, 2020
SilverG33k:
Honestly, let's just catch cruise here.

Now here's a quick question, you're to build nairaland backend, how would you do that?

What technologies would you use, how would you handle scalability issues. Note than, the expected traffic for this system is 1k req/s just for fun tho

No yabbing

In my case I would use a decoupled architecture. That is, separate the front-end from back-end and host them separately.
Front end would be an Angular app hosted on firebase, while back-end would be a Django Rest API hosted on Google App Engine with auto-scaling to handle traffic surges.
I'd utilize a caching system like Memcached to reduce load on the back-end server, store the generated contents like media files on a separate file storage service like Amazon S3 or Google Cloud Buckets, host the database separately on Google Cloud SQL, and finally use a CDN like Cloudflare.
This setup would be expensive though, but for a platform that gets such a huge amount of requests I'm certain the owner would be able to afford it.

88 Likes 13 Shares

DC49: 6:41am On Dec 06, 2020
DC49:


In my case I would use a decoupled architecture. That is, separate the front-end from back-end and host them separately.
Front end would be an Angular app hosted on firebase, while back-end would be a Django Rest API hosted on Google App Engine with auto-scaling to handle traffic surges.
I'd utilize a caching system like Memcached to reduce load on the back-end server, store the generated contents like media files on a separate file storage service like Amazon S3 or Google Cloud Buckets, host the database separately on Google Cloud SQL, and finally use a CDN like Cloudflare.
This setup would be expensive though, but for a platform that gets such a huge amount of requests I'm certain the owner would be able to afford it.
Karleb(m): 7:20am On Dec 06, 2020
Decoupled architecture. Noted!

@Mukina2 This deserves front page

1 Like

SilverG33k(m): 5:16pm On Dec 06, 2020
DC49:


In my case I would use a decoupled architecture. That is, separate the front-end from back-end and host them separately.
Front end would be an Angular app hosted on firebase, while back-end would be a Django Rest API hosted on Google App Engine with auto-scaling to handle traffic surges.
I'd utilize a caching system like Memcached to reduce load on the back-end server, store the generated contents like media files on a separate file storage service like Amazon S3 or Google Cloud Buckets, host the database separately on Google Cloud SQL, and finally use a CDN like Cloudflare.
This setup would be expensive though, but for a platform that gets such a huge amount of requests I'm certain the owner would be able to afford it.

I love this, interesting, you would definitely be needing devops. Great thinking really

1 Like

SilverG33k(m): 5:35pm On Dec 06, 2020
What if you we try to reduce cost by

Using just AWS, or a good old server, right, host the database on a separate server, and the other contents(images, videos, profile pictures), serve the media files with cloudflare or AWS cloudfront, then in cases where the homepage gets accessed by thousands of s, a Memcached would work, I prefer to use Redis however, both does the same thing, so instead of reading the popular topics and posts off the database, it reads from the cache, which reduces load on the database server.

For analytics, i always prefer and suggest to use a nosql db like mongo db, cassandra or faunadb, anything flexible actually, obviously our core database is going to be a relational database e.g mysql, oracle so we don't want to put too much load on it, we just need to put in the critical data there, for scalability, so this means that all of those number of views, daily active s, page views, e.t.c would go to a nosql database, s logs as well, requests and responses(if you're keeping track) all of these analytical data should make use of firebase database or any nosql db you want

Am i missing anything, yes, anytime i , I'd share, this is how i am going to build mine, its not a general rule that everyone must follow sha, let us understand that

18 Likes 1 Share

SilverG33k(m): 5:38pm On Dec 06, 2020
talk2hb1:
Tooh,
Waiting for Experience Bosses to Come Tutor us on this.

lmao look at this boss oh, chai

Okay, Sir... How would you handle 10 backend devs working with 4 different languages
talk2hb1(m): 5:42pm On Dec 06, 2020
SilverG33k:


lmao look at this boss oh, chai

Okay, Sir... How would you handle 10 backend devs working with 4 different languages
This sha want to intimidate me 10backend Dev, 4 different languages.
I go dey form big boss ni ooo grin

2 Likes

tensazangetsu20(m): 5:43pm On Dec 06, 2020
DC49:


In my case I would use a decoupled architecture. That is, separate the front-end from back-end and host them separately.
Front end would be an Angular app hosted on firebase, while back-end would be a Django Rest API hosted on Google App Engine with auto-scaling to handle traffic surges.
I'd utilize a caching system like Memcached to reduce load on the back-end server, store the generated contents like media files on a separate file storage service like Amazon S3 or Google Cloud Buckets, host the database separately on Google Cloud SQL, and finally use a CDN like Cloudflare.
This setup would be expensive though, but for a platform that gets such a huge amount of requests I'm certain the owner would be able to afford it.
I am building something very similar. The frontend is React. The backend is WordPress.
Fash20: 3:23am On Dec 07, 2020
Good... I love to learn from booses... I haven't gotten the chance to work with or learn from one directly but I love what I have read here so far
cixak95211: 6:58am On Dec 07, 2020
SilverG33k:
Honestly, let's just catch cruise here.

Now here's a quick question, you're to build nairaland backend, how would you do that?

What technologies would you use, how would you handle scalability issues. Note than, the expected traffic for this system is 1k req/s just for fun tho

No yabbing

NL say she has 2.5M , if that is anything to go by then indeed it has to be a microservice. There must be very bad guys amongst these , so an API Gateway is a must for the backend. I'll likely go with Kubernetes so it handles upscaling and downscaling for me without having me loose sleep. NGINX is very much ok as an LB for both frontend and backend, i prefer a round-robin as against weighted since the K8 engine will spin up equally weighted infra. For the DB, i'll likely go with Mongodb since it will allow me run a search engine without the need for Algolia . . oh well Algolia can come later , but at least I can do database sharding right out of the box. You dont expect me to dump 2.5M parent records which will span into hundreds of millions, imagine one has 100 posts, which is way likely to be more each week. Fact is i cannot dump those in one single DB, no maatter hwo powerful it is, It has to be redundant. Redis is needed for caching frontpage topics, so i dont have to serve it all the time, for each request. I could cache with NGinx too but i prefer redis. LRU is powerful.
Choice of server infra . . a simple linux image will do, ssd based and most importantly, all servers, FE, BE and DB must be encrypted at rest. I'll use a KMS for that. I'll leave one bastion host available for maintenance work and make sure all servers only communicate via a VPC.

17 Likes 2 Shares

SilverG33k(m): 8:02am On Dec 07, 2020
cixak95211:


NL say she has 2.5M , if that is anything to go by then indeed it has to be a microservice. There must be very bad guys amongst these , so an API Gateway is a must for the backend. I'll likely go with Kubernetes so it handles upscaling and downscaling for me without having me loose sleep. NGINX is very much ok as an LB for both frontend and backend, i prefer a round-robin as against weighted since the K8 engine will spin up equally weighted infra. For the DB, i'll likely go with Mongodb since it will allow me run a search engine without the need for Algolia . . oh well Algolia can come later , but at least I can do database sharding right out of the box. You dont expect me to dump 2.5M parent records which will span into hundreds of millions, imagine one has 100 posts, which is way likely to be more each week. Fact is i cannot dump those in one single DB, no maatter hwo powerful it is, It has to be redundant. Redis is needed for caching frontpage topics, so i dont have to serve it all the time, for each request. I could cache with NGinx too but i prefer redis. LRU is powerful.
Choice of server infra . . a simple linux image will do, ssd based and most importantly, all servers, FE, BE and DB must be encrypted at rest. I'll use a KMS for that. I'll leave one bastion host available for maintenance work and make sure all servers only communicate via a VPC.

Database sharding how did I miss that... Excellent point there, very necessary, why would the DB be encrypted tho? Or what part of the DB needs to be encrypted, please share, because I am just considering the speed and complexity

1 Like

Karleb(m): 8:10am On Dec 07, 2020
FE, BE, VPC, KMS
What do they mean?
Amoto05(m): 12:34pm On Dec 07, 2020
k
Dottore: 12:35pm On Dec 07, 2020
Ok
Jahzrockballer(m): 12:36pm On Dec 07, 2020
Just curious, why is no one talking about ruby on rails Just saying though

2 Likes

Lovelies1: 12:36pm On Dec 07, 2020
cinoedhunter: 12:37pm On Dec 07, 2020
Please what language are you guys speaking here sad undecided cryayam not understanding

19 Likes 1 Share

Olalekank(m): 12:38pm On Dec 07, 2020
.
JoeEeL(m): 12:40pm On Dec 07, 2020
Good. These are the kind of threads that, once in a while, should be popping up on front page and not ridiculously silly-ass posts like Regina daniels prevents Ned's 7th wife, Toke Makinwa stuns with plastic-ass ...

Backend, as strictly relates to Web architecture, NodeJS is my go-to. Of course with Express as the de-facto server framework. Request / sec, as far as node is concerned is nearly 2x, response time is as much as 35% faster. Also it abstracts quirks, inconsistencies and is the perfect syntactic sugar.

Looking to annex its powers to FE, one has to be thinking in of React -- which leads you to the promised land -- THE MERN stack

React helps you define the UI once and for all. Then, when the state of the app changes, the UI is rebuilt to react to the change -- acting on its name.
Small manageable components enable to build fairly complex apps but the sweetness of this is, you're still running on vanilla JS anyways. DOM node-hunting is kept at barest minimum; an avoidance of pain you might argue but darn! I was good at nifty DOM selection in my jQuery heydays. Anyways, all you do is maintain the state of your app (with a regular old JavaScript object) and the rest just naturally follows.

18 Likes

cixak95212: 12:41pm On Dec 07, 2020
SilverG33k:


Database sharding how did I miss that... Excellent point there, very necessary, why would the DB be encrypted tho? Or what part of the DB needs to be encrypted, please share, because I am just considering the speed and complexity

2 Likes

ipressplay: 12:44pm On Dec 07, 2020
DC49:


In my case I would use a decoupled architecture. That is, separate the front-end from back-end and host them separately.
Front end would be an Angular app hosted on firebase, while back-end would be a Django Rest API hosted on Google App Engine with auto-scaling to handle traffic surges.
I'd utilize a caching system like Memcached to reduce load on the back-end server, store the generated contents like media files on a separate file storage service like Amazon S3 or Google Cloud Buckets, host the database separately on Google Cloud SQL, and finally use a CDN like Cloudflare.
This setup would be expensive though, but for a platform that gets such a huge amount of requests I'm certain the owner would be able to afford it.

This like you said is very expensive.

Why not build with Laravel, its scalable and enjoys massive .

Hosting with VPS or dedicated server. Better still use AWS for hosting and different buckets for different files.

My thoughts.

7 Likes

SenorFax(m): 12:46pm On Dec 07, 2020
Please guys, start a new one for learners like us. I love the languages you are speaking, but I'm a newbie to software techs

4 Likes

ElRapido: 12:47pm On Dec 07, 2020
I dey laugh here
kennyflames(m): 12:49pm On Dec 07, 2020
k
Opexzy: 12:49pm On Dec 07, 2020
I will come back to give my opinion...
benedictuyi(m): 12:52pm On Dec 07, 2020
Following
WiszyFraud: 12:52pm On Dec 07, 2020
cool
*Tightens seatbelt*

Time to learn

1 Like 1 Share

(1) Reply)

Free Arduino Tutorials

(Go Up)

Sections: How To . 53
Disclaimer: Every Nairaland member is solely responsible for anything that he/she posts or s on Nairaland.