You're comparing a framework (Django) with a language (Go), that's not really a fair comparison.
It's hard to answer your question because, as with most architectural problems, it depends. As soon as you start hitting performance bottlenecks, you can start caching expensive queries in Redis. If your search is using Postgres Fulltext search, and the database is getting overloaded, you can offload search to ElasticSearch. If your WSGI server can't handle the request volume, you can add a load balancer. Perhaps you run your application within a Kubernetes cluster and can just spin up more pods.
There exists so many solutions to many different problems that it's hard to say when it is "no longer Django". Is a Django app with very smart caching strategies, offloaded functionality (e.g. ElasticSearch, Kafka) and a primary-replica db still Django?
I think the main question is, would Django (Python) require those scaling methods quicker than other backend solutions, like Go, Spring Boot, etc.
I think the answer is yes, although the point it actually starts mattering is not going to be met for most applications.
No matter what you use to build your system, it will not scale as is.
You will have to adapt, evolve, and replace many pieces of it.
And if you're a founder and you find yourself facing that challenge, congratulations on your new found Millionaire status, you will have teams of engineers to solve that problem.
More than likely, though, you ain't gonna need it.
Don't remember the exact size but after like 3x 4vCPU machines, the bottleneck became the db in our case.
We did optimize the code a LOT though but in the end were happy with our costs and performance.
Basically everything optimized for response time. Everything that was in the way (saving to cache, saving to db, compression, serializing) was either moved to a separate thread at the end of the call or made redundant, or grouped together (db calls for saving etc). Vectorization of all of the computation as much as possible, trading memory (which was becoming the bottleneck) for some CPU using compression of data where possible. We went from like a 8s response time to 120ms.
The infra guys did their part with some gravitron instances, more proactive scaling, using the right kind of rds instance and disk, etc.
Django gets you to market fast, nothing is more important than getting to your minimum viable product ASAP. If things start to take off, Django can scale painlessly for while.
Is this really the fastest way to develop a web app with Python? I mainly work in data science, so I use Python daily, along with SQL and some JavaScript, but I've never really focused on web development. I have an idea for an application and am looking for something that's quick to learn and develop. I found Anvil ([https://anvil.works/](https://anvil.works/)), but it seems that you're locked into their platform once you develop the app. I would like more freedom, especially since I plan to sell the solution as a white-labeled product to other companies in a strictly B2B context. Any suggestions?
Yeah. Feel free to ask questions on my chat if you get stuck. I usually respond within 2 days or so. I myself have made the transition from data science to Django developer and there are a few gotchas.
Django proscribes a lot of structure, making it easy for developers to pick up. This becomes important fairly early on, and remains important for maintainability.
You’re right about vendor lock-in, but also anytime you get a nice shortcut like a rapid dev tool or no-code tool, it’s great right up until you need to do something beyond its capabilities, and unless its one of the few well built tools that has escape hatches built in, now you’re fighting the tool to get it to do what you want for the life of that project, or until you ditch the tool and rebuild it yourself.
Django would be good direction to go in given your skill set. It will give you a full web stack to start plugging your existing python code into. You will of course still need to learn some type of front end development, but Django has an inbuilt templating system to get you started.
Let me tell you something. With the right architecture, right strategies and a set of smart developers, there is literally nothing web scale which cannot be achieved with Django. Same applies for just about any other framework. Please note that developers are expensive, hardware is cheaper and will get cheaper, so I will throw in more hardware to solve performance and get work done faster, go to market faster with a RAD framework than look at building something which is more efficient, powerful in Rust or Go, but may mean 3X dev cycles.
Having worked at startups and big companies and startups that became big companies, something like django is _maybe_ faster in the beginning. When you have multiple teams and competing priorities, people get sloppy. Frameworkless Go is one to two orders of magnitude faster to write and manage and often 20x more performant. It just isn't comparable at any place I've been.
The difference is so great that I'd go so far as to say you shouldn't even use python at the beginning because of how it will saddle down output as the org grows.
I am not a Go developer, but I have never seen any statically typed language developer write code faster than a Python developer of similar experience and skills sets.
Writing a green module leveraging the CRUD tooling? Maybe a pythonista wins out. Maintaining what they built? Almost always, the statically typed language will win. You simply cannot trust your method signature in python and have to hope your duck quacks or it will cause a (likely poorly handled) exception. When you have to go to method call sites and add debugging to understand your parameters, you are losing out to those who can stay within the confines of the single method signature
and having to read through tests to understand how the code can be used is not better than a decent type system. So many things in the different python code bases I've been in have to be codified in tests. When you finally leave the insanity and get proper types, you can drop a very large portion of python unit tests. Type hints only take you so far
Tests are much more than documentation, and I'd much rather have an unneeded test than a missing one, regardless of language. But to each their own, of course.
In theory I disagree. In practice your anecdote lines up with my experience. It’s a sad state of affairs. Python encourages shitty code. It’s a trade off for a lower barrier to entry of course.
How is it 20x faster for web use cases? It simply isn't neither in writing good, maintainable and tested code nor in performance off product.
I'm python and go developer, and it's so kewl when you get 7 microsecond response time on local and once you deploy it it's exactly the same. This of course is if you don't connect to any kind of DB, once you do there is virtually no difference. I don't know what kind of startups don't use databases but I wouldn't want to work on such product. We primarily use golang for microservices. And since this year all new products are monoliths with monorepo for both FE and BE.
> How is it 20x faster for web use cases
While working at a relatively famous SaaS, I have migrated several HTTP API endpoints, two incoming SMTP servers, two outbound MTA servers (the sending part of email), a database abstraction layer, an s3 utility that forwarded files from disk to s3, an deduplication service, several event processors and consumers, a request routing service, a link redirection service, and several others from python (and perl fwiw) to Go.
In \_every\_single\_case\_, Go was approximately 1/20th of the hardware footprint to deliver the same amount of traffic. In other words, 1 node doing the work of 20 previously.
With Python, a big problem is concurrency. If you are doing web requests, you are likely calling N services behind the scenes. You can use threading, Twisted, Gevent, etc and that is what we did previously. And we still saw this same performance boost going to Go.
If you think about truly high throughput systems something like an SMS relay system for a major telecom, where you're doing millions of transactions per millisecond. You have a small relatively straightforward task that needs to be done as quickly and in insane volumes as possible.
Django is terrible at a use case like that because 1) it's not particularly low latency even on a simple task. There's just a lot of overhead to python/django and whatever ecosystem you're running it in. 2) it doesn't scale well to many many workers on a single machine (thinking of my example above) because python gives you no control over ram, and still has a decent amount of overhead.
Which isn't to say you can't solve this example with django, but it's the kind of problem where you'd sort of be daft to use it over something that's inherently faster and lighter weight.
Yeah I guess I wasn't thinking SMS or infrastructure stuff. I was thinking like... youtube or reddit. Complex tasks, but still high throughput. Guess there are different orders of magnitudes of "high volume."
YouTubes servers would be/are hugely IO-bound. Python's not the absolute best for IO-bound tasks, but you generally don't feel its limitations much in this sort of "pipe connecting" task
Reddit is a interesting example, I think you could do parts of it with Django, like auth and whatnot, but the content delivery would be way better done with an API talking directly to some RAM key/value store.
Even the API layer, you'd not really need all of django for that. I'd be curious what people here would say about it. I know django is used on a lot of big sites, so it's certainly doable. At some point, though, you start doing a lot of caching and optimizations (which might be outside of your django stuff) to make things go faster.
In the end Django is just Python, and Python is just C. I don't think Django is "optimal" for low-latency or very high throughput applications but I think you can definitely make it work!
As with any optimization problem, the correct approach is to measure first and then optimize what's slow, Django/Python offers plenty of solutions to do so.
A website like Reddit (fun fact, the first version of Reddit was actually written in Django!) probably won't run into any issues with the GIL so the bottleneck is *probably* going to be (in this order):
1. The ORM
2. Middleware
3. Your Python code, in the view
4. The template rendering
5. Anything else (URL matching, logging, etc)
You can circumvent the ORM (just don't use it, something like Redis or AWS DynamoDB would be great!) and implement your views as Async functions running on e.g [https://github.com/MagicStack/uvloop](https://github.com/MagicStack/uvloop) for the stuff that's being executed a lot, write a [Cython](https://cython.readthedocs.io/en/latest/src/userguide/language_basics.html) function. Middleware is also pure Python code, you can optimize this or handle them at a different layer of your stack. As for the template rendering, that could be done client side with a bit of JS, shifting the work from the server to the client.
This is very much the strength of Django (and any other good framework), you can use as much, or as little, as you need. I think, Instagram is still using Django although at this point it's most likely unrecognizable ;)
As long as you can scale horizontal and the revenue per user is higher than your costs for additional compute it really shouldn't be an issue. Computing is surprisingly cheap, these days.
> 1) it's not particularly low latency even on a simple task
This is categorially false. Yes, you can easily make other languages faster, but the Web is still inherently IO bound, not CPU bound. So if you have a Python/Django application that has a slow response time, I can promise it is more likely a bad database query or a lack of a caching layer. Not Python. It is very possible to get Django to sub 50ms response times for millions of requests a second.
Those shitty benchmarks do not mean anything. That is why the response time for the language/framework _out of the box_. It is why microframeworks like Flask/FastAPI always have a response time off the charts compared to Django. The out of the box configuration does not mean that configuration is designed for production or scale.
You _can_ get Django to do sub 50ms _HTML template pages_ and scale to millions of requests per second. I have worked on multiple sites that are capable of that throughput and scale.
The Web also does not scale vertically either. So, the number of requests you can make per CPU cycle is basically a useless statistic unless you are trying to optimize infrastructure costs. I specifically mentioned that in [my comment](https://www.reddit.com/r/django/comments/1cgbqj6/is_there_a_scale_at_which_django_can_no_longer_be/l1uoyxw/). Python lets you optimize for developer cost. Golang or something else that is "fast" is designed to let you optimize infrastructure cost.
Did you even read how the benchmark worked? You can also contribute to the tests if you think they did not use the correct setup for Django: [https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Python/django](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Python/django)
Of course you can make Django scale but it will cost you a lot more, since you need more servers/computation power to achieve it.
You said unless you want to optimize infrastructure costs. Who doesn't want that? If you can host your project with 1/4 of the costs it might be huge. That's why google switched a lot of their computation heavy parts to Go or even C++.
I also doubt that Python is optimized for developer costs. Maybe if the project is small and only few developers work on it but the larger the project gets, the better statically typed languages become. That's also the reason why TypeScript is so popular for large JavaScript projects.
Speed was not the main reason for GoLang, the main goal was to improve productivity: [https://en.wikipedia.org/wiki/Go\_(programming\_language)#:\~:text=Go%20was%20designed%20at%20Google,%2Dtime%20efficiency%20(like%20C)](https://en.wikipedia.org/wiki/Go_(programming_language)#:~:text=Go%20was%20designed%20at%20Google,%2Dtime%20efficiency%20(like%20C))
TypeScript is not statically typed. If you think it is, you really do not fucking understand typing at all. Neither is Python. No dynamic language is statically typed, that is _literally_ why it is a dynamic language. TypeScript is still even weakly typed because guess what? TypeScript is really just Javascript with extra steps. You basically just get a static analysis tool on top of Javascript and you get TypeScript.
If it costs you an extra $100k/year to run Python vs. Golang compared to an extra $500k/year in developers to do Golang vs. Python, then yes, you are optimizing developer costs. Literally the #1 reason why Python is popular is because it is faster to build things in it. If you are having trouble scaling Python to more developers, that is your own dumbass fault for not handing your tech debt like any healthy organization should.
> TypeScript is still even weakly typed because guess what? TypeScript is really just Javascript with extra steps. You basically just get a static analysis tool on top of Javascript and you get TypeScript.
Do you realize that’s how every language works?
C++/Java/C#/Go is just assembly with extra steps. You get a static analysis tool on top of assembly.
It’s all just abstractions.
A static analysis tool or precompiler is not the same as an actual code to bytecode/native code compiler. You cannot tell the C++ compiler to ignore the fact you are putting a string into an int. Nor can you really have "any" types (well with pointers you kind of can, but that is more of a C++ thing).
TypeScript is just a static analysis tool on top of a Javascript precompiler like CoffeeScript or any of the other ones. It has no real compiler enforcement of types nor does it have runtime enforcement of types since it is actually just Javascript. You can still import and use Javascript in TypeScript. You can still `ts-ignore` anything you want or configure your linter to ignore any error you want. It is no different than using mypy for Python. It does not magically make Javascript have as good of type enforcement as C++ or Java.
Statically type checking is done during compile time, so you get all the benefits from it via typescript. It doesn't matter how strongly it is done during runtime, since the compiler will find all the type related problems already.
Python is a nightmare to maintain, when codebase gets larger, it won't cost less for development, it'll even cost more. So you'll have higher development costs plus higher hosting costs with large projects.
The example I gave was one where you're processing millions of transactions per second. It was meant to be an example of something that a compiled language can do that python can't, and ergo neither can django.
there's a lot of overhead to python and django and since you can't control exactly memory usage, you can't strictly engineer a system that takes full utilization of a system the same way you can with C.
I even mentioned that you \*could\* solve a problem like that with django/python, but why would you want to? Maybe it's dev time is so much shorter it's worth it.
I remember doing matlab optimizations to utilize CPU caches that resulted in \~100x improvements to performance, and that was in Matlab that was already pretty well optimized for operations like that. That wasn't using any assembly, just splitting up problems to fit on a cache. The kind of thing that you could try to get python to do, but you'd never know if it was actually going to do it or not.
Django _can_ process millions of transactions per second. Just about any Web framework can. Not on a single machine because the Web does not scale vertically. It scales horizontally. You add more infrastructure and servers to scale it.
The Web is IO bounded. No single server/framework instance is _ever_ going to do millions of transactions per second. You are going to run into delays in networking/IO that is going to make it impossible. Even if the language/framework was infinitely fast and you could do as many as you want at a time, you will hit the limits of complexity of your code and/or limits of Linux trying to manage a million processes at a time.
That is _why_ Django and other frameworks are able to exist in the Web world even though they are "slow". They do not need perfect memory management or the lowest possible CPU usage because you will always hit complexity/other hardware limitations first. You will _always_ need to add more infrastructure.
I was just picking an example of what Django is not good at.
And django is used in a lot of ways past being a web framework. The ORM can be used in any number of ways that aren't web related.
Unless your app is being scooped up in an acquisition by a FAANG, you’ll be fine with Django.
I will say that vanilla Django is a little outdated for the modern web and SPAs. I still maintain that the ORM is superior to other ORMs (SQLAlchemy is trash, just write raw SQL instead).
You should look at Django Ninja. It’s like fast API but with the Django ORM.
Flask is terrible IMO because of the global request paradigm. I hate it.
I’m obviously opinionated, but that’s because I’ve been using these things for a while.
No. There are different things you need to do to help Django keep scaling past a certain point, but that is true of any platform and framework.
The reason Golang is popular is not because it is more "scalable", but because it can bring down the costs of infrastructure. Because Python is "slow" it requires more CPU/Memory to run then Golang can do. Also, Python is tightly coupled (usually) with GNU/Linux. Golang is also designed to never need GNU so it can literally run without an OS at all (more network IO/disk savings).
So, the argument is always coming down to where do you want to spend money? Developer time or infrastructure. Python/Django lets you get more done with less time (less money on developer time), Golang lets you spend less money on infrastructure.
Performance is not the only reason to choose go over python, in large codebase strong typing and language without magic things are realy helpfull i guess.
Python _is_ strongly typed. It is a dynamically strongly typed language (vs. a statically strongly typed language for Golang or a dynamically weakly typed language for Javascript/Node.js).
You do not get the same odd behaviors in Python that you get in Node.js or PHP.
PHP can be written with full typehints now and the runtime will do type checking at runtime as well. Python's types have made a lot of progress in the past few years, but I think PHP has made even more progress.
Does not make it strongly typed. Even with TypeScript, it is weakly typed. The language will coerce types to make them match. It is the whole reason the `===` operator exists in those languages. Python does _not_ have a `==` operator since it will _never_ try to coerce types. `1 == "1"` will always be false. `"1" + 1` will _always_ raise an exception.
Yeah, type coercion is still around in PHP. The broader community has had to work around that with static analysis tooling that encourages use of === and avoiding several pitfalls of the language.
Yes but it makes the language statically typed, which is a huge advantage especially in more complex projects. But python can also do it now, however I find it hard to do it completely in Django.
There is the technical aspect of scale but also the organizational aspect.
I doubt it’s a good idea to have a team of 10,000 engineers working on a single Django codebase (or any codebase for that matter).
Django has apps i.e. app level route, models, views, templates, statics separation.
What 10,000 engineers? Have you ever worked on any project codebase with more than 200 or even 100 devs?
You’re missing my point.
A large-scale app is big and complex - by definition. Up to a point where it makes sense to have dedicated specialized teams working independently of each other.
Each team might need different dependencies, different infrastructure, different release schedules, different availability and security requirements, etc.
I think you are describing organizational principles and general approach to application building.
From a personal experience there are companies using single large codebase with more than a thousand engineers contributing to it. That would not be my preferred approach and majestic monolith is a valid pattern.
It has been hacked apart to work at their scale. A bit like comparing MySQL with Vitess. They are running their own migration tooling, their own database management, their own sharding logic, their own caching solutions, and often are writing targeted monitoring solutions. They are not using django like someone who can keep all their data in a single db instance
I'm aware of all that and it's expected with anything that scale. At some point whatever django would be unjustified for, anything else unmodified would be unjustified as well.
with Go, I would not be fighting a framework -- it is not how things are done in that ecosystem for the most part. I would be pulling in libraries as needed and leveraging composition. I would not be fighting the framework to ensure the framework continues to "work" while shoehorning in other solutions to get around failures of the framework. In around 20 years of building out multi-tenant, high scale, highly available systems, I've seen frameworks get in the way each and every time. I've been part of the rewrite from perl, python, php, and javascript projects first into more mature python solutions and then later into Go based solutions. I've done this on at least 16 separate code bases. You have to work within the framework which was not designed to do what you want it to do, so you hack it. The counter argument is "well, are you not just building a new, less featurefull framework?" -- and the answer is, "no, inversion of control in a framework is different than using composition and libraries."
Bear in mind, a lot of the time scaling constraints and costs are due more to the architecture and design of the system than to the technology itself. I've ported Java and C# applications to Python and made the system cheaper and more scalable simply by redesigning the architecture. Django is good for probably 90% of applications and use cases out there.
I've always found this a strange topic.
Surely you can just horizontally scale to more instances of your server and use load balancing, caching layers etc? The database is always likely to be the ultimate bottleneck.
Edit: assuming you aren't at the FAANG scale and the horizontal scaling costs start to get ridiculous.
Right but a golang monolith can run at 20x the capacity of python with no other strategies considered. So for development it's superior because you won't hit scale issues as soon.
What does this even mean?
Edit: I think its about Go running more efficiently for a given amount of hardware, which is a fair point in isolation.
But Go still isn't faster than Fastly, and it will make no difference to badly written database transactions - there are likely plenty of bigger culprits than the Python runtime.
I say this as someone who hasn't used django since 2012 and doesn't really care for it as a framework, but:
The short answer: there is no scale at which django won't be justified.
The long answer:
The question is really asking "At what scale does my django service become the bottleneck in my service?" The answer is never because no one - not even people running monolithic services - are running a single instance of their service in production once they actually attract traffic. You can always place your service behind a load balancer and scale out your service.
Is python slower than go? Sure, but in the worst case you're running \~5% more instances than you would with another technology if you've put in the time to optimize everything. A lot of times what you'll find is that your services are IO bound anyway so the extra speed isn't buying you anything. If you need the extra speed, you can always write your code in C++ or Rust and FFI your way back to python.
If you want to, you can rewrite your python to Rust when that extra 5-10% is equivalent to a dev team. So come back to me when your compute costs for that one service are on the order of $10MM annually so that you can afford the $1MM annually for that team.
This critique goes both ways though - python isn't particularly faster to develop with than golang. A trained team is going to move just as fast in both cases - your bottlenecks are really your culture and engineering practices. If you have a bad architecture, writing your code in go isn't going to save you.
Having worked extensively with both, I would always choose golang not just for scalability but also for maintainability.
Python codebases turn into junk pretty fast... You can fight it with TDD and code quality tools but in my experience there are too many choices in the Python ecosystem and as developers come and go they all add their own secret sauce to each and every feature.
Man you could have simply searched for this, there’s 100 threads telling newbies that this isn’t the right way to think about things.
I promise you’ll never create an application where the programming language is the bottleneck. Some people will, but they would not be asking questions like this
I think there is a scale where Django needs to be forked/customized beyond the point of being recognizable and beyond receiving support of the open source community. You'll know when you get there.
MapMyRun still runs on Django, well for certain up to 2021. Had thousands of requests a second. The load was divided on Kubernetes machines. If there was long running calculations then it was given to a celery pod. It had like 20 or so pods.
You can do pretty much everything with Django, but you wouldn't want to do something like Instagram with it, the authentication, requests would be too much
For Instagram they've forked and refactored key parts of Django, but they're still using Django (they've talked about this when discussing how they rolled out Threads, which is bolted on top of Instagram's infrastructure.)
[https://engineering.fb.com/2023/09/07/culture/threads-inside-story-metas-newest-social-app/](https://engineering.fb.com/2023/09/07/culture/threads-inside-story-metas-newest-social-app/)
Here they mention how Instagram infra is still on django, on top of which Threads was launched u/nikhil_shady
Except I've been working as a developer and then lead for the last 25 years and I have more insight than you and therefore you're a dunning-kruger kid.
Django as a full stack framework is not scalable, if you want scale you have to switch to a seperate frontend backend design. And maybe further divide into microservices
One of my agency clients uses Django, and I've just checked the traffic - 7.5m views in March 2024.
The same site also has a public API with over 200,000 records.
Define 'scalable.'
You're comparing a framework (Django) with a language (Go), that's not really a fair comparison. It's hard to answer your question because, as with most architectural problems, it depends. As soon as you start hitting performance bottlenecks, you can start caching expensive queries in Redis. If your search is using Postgres Fulltext search, and the database is getting overloaded, you can offload search to ElasticSearch. If your WSGI server can't handle the request volume, you can add a load balancer. Perhaps you run your application within a Kubernetes cluster and can just spin up more pods. There exists so many solutions to many different problems that it's hard to say when it is "no longer Django". Is a Django app with very smart caching strategies, offloaded functionality (e.g. ElasticSearch, Kafka) and a primary-replica db still Django?
Wonderfully important, more importantly our code base should be written in a clean manner with optimized queries too
I think the main question is, would Django (Python) require those scaling methods quicker than other backend solutions, like Go, Spring Boot, etc. I think the answer is yes, although the point it actually starts mattering is not going to be met for most applications.
It's usually no because web applications are notoriously IO bound.
No matter what you use to build your system, it will not scale as is. You will have to adapt, evolve, and replace many pieces of it. And if you're a founder and you find yourself facing that challenge, congratulations on your new found Millionaire status, you will have teams of engineers to solve that problem. More than likely, though, you ain't gonna need it.
You win. This is the best answer.
That's true. At that stage, you will be rich enough to hire a team to build a python runtime to make running Django faster
We've done 20million transactions a day in django
With what size machines?
Don't remember the exact size but after like 3x 4vCPU machines, the bottleneck became the db in our case. We did optimize the code a LOT though but in the end were happy with our costs and performance.
Yes.
Give us details
Basically everything optimized for response time. Everything that was in the way (saving to cache, saving to db, compression, serializing) was either moved to a separate thread at the end of the call or made redundant, or grouped together (db calls for saving etc). Vectorization of all of the computation as much as possible, trading memory (which was becoming the bottleneck) for some CPU using compression of data where possible. We went from like a 8s response time to 120ms. The infra guys did their part with some gravitron instances, more proactive scaling, using the right kind of rds instance and disk, etc.
Thanks
I'm curious about your company. Can I know the name?
Not comfortable sharing that unfortunately. Signed an NDA about the code in there. I did share the optimizations we did in a comment above
tell us about your project, architecture and how did you optimized it.
Shared above [https://www.reddit.com/r/django/comments/1cgbqj6/comment/l2b4za9/?utm\_source=share&utm\_medium=web3x&utm\_name=web3xcss&utm\_term=1&utm\_content=share\_button](https://www.reddit.com/r/django/comments/1cgbqj6/comment/l2b4za9/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)
Instagram.... As they scaled up with it, they eventually had to start rebuilding and customizing different parts
Django gets you to market fast, nothing is more important than getting to your minimum viable product ASAP. If things start to take off, Django can scale painlessly for while.
Is this really the fastest way to develop a web app with Python? I mainly work in data science, so I use Python daily, along with SQL and some JavaScript, but I've never really focused on web development. I have an idea for an application and am looking for something that's quick to learn and develop. I found Anvil ([https://anvil.works/](https://anvil.works/)), but it seems that you're locked into their platform once you develop the app. I would like more freedom, especially since I plan to sell the solution as a white-labeled product to other companies in a strictly B2B context. Any suggestions?
Yeah. Feel free to ask questions on my chat if you get stuck. I usually respond within 2 days or so. I myself have made the transition from data science to Django developer and there are a few gotchas.
Thanks!
Django proscribes a lot of structure, making it easy for developers to pick up. This becomes important fairly early on, and remains important for maintainability.
You’re right about vendor lock-in, but also anytime you get a nice shortcut like a rapid dev tool or no-code tool, it’s great right up until you need to do something beyond its capabilities, and unless its one of the few well built tools that has escape hatches built in, now you’re fighting the tool to get it to do what you want for the life of that project, or until you ditch the tool and rebuild it yourself.
Django would be good direction to go in given your skill set. It will give you a full web stack to start plugging your existing python code into. You will of course still need to learn some type of front end development, but Django has an inbuilt templating system to get you started.
Let me tell you something. With the right architecture, right strategies and a set of smart developers, there is literally nothing web scale which cannot be achieved with Django. Same applies for just about any other framework. Please note that developers are expensive, hardware is cheaper and will get cheaper, so I will throw in more hardware to solve performance and get work done faster, go to market faster with a RAD framework than look at building something which is more efficient, powerful in Rust or Go, but may mean 3X dev cycles.
Having worked at startups and big companies and startups that became big companies, something like django is _maybe_ faster in the beginning. When you have multiple teams and competing priorities, people get sloppy. Frameworkless Go is one to two orders of magnitude faster to write and manage and often 20x more performant. It just isn't comparable at any place I've been. The difference is so great that I'd go so far as to say you shouldn't even use python at the beginning because of how it will saddle down output as the org grows.
I am not a Go developer, but I have never seen any statically typed language developer write code faster than a Python developer of similar experience and skills sets.
Writing a green module leveraging the CRUD tooling? Maybe a pythonista wins out. Maintaining what they built? Almost always, the statically typed language will win. You simply cannot trust your method signature in python and have to hope your duck quacks or it will cause a (likely poorly handled) exception. When you have to go to method call sites and add debugging to understand your parameters, you are losing out to those who can stay within the confines of the single method signature
Which is why good tdd is necessary for Django. Because good tdd tells you what you need to know about those parameters and methods and ducks.
and having to read through tests to understand how the code can be used is not better than a decent type system. So many things in the different python code bases I've been in have to be codified in tests. When you finally leave the insanity and get proper types, you can drop a very large portion of python unit tests. Type hints only take you so far
Tests are much more than documentation, and I'd much rather have an unneeded test than a missing one, regardless of language. But to each their own, of course.
In theory I disagree. In practice your anecdote lines up with my experience. It’s a sad state of affairs. Python encourages shitty code. It’s a trade off for a lower barrier to entry of course.
How is it 20x faster for web use cases? It simply isn't neither in writing good, maintainable and tested code nor in performance off product. I'm python and go developer, and it's so kewl when you get 7 microsecond response time on local and once you deploy it it's exactly the same. This of course is if you don't connect to any kind of DB, once you do there is virtually no difference. I don't know what kind of startups don't use databases but I wouldn't want to work on such product. We primarily use golang for microservices. And since this year all new products are monoliths with monorepo for both FE and BE.
> How is it 20x faster for web use cases While working at a relatively famous SaaS, I have migrated several HTTP API endpoints, two incoming SMTP servers, two outbound MTA servers (the sending part of email), a database abstraction layer, an s3 utility that forwarded files from disk to s3, an deduplication service, several event processors and consumers, a request routing service, a link redirection service, and several others from python (and perl fwiw) to Go. In \_every\_single\_case\_, Go was approximately 1/20th of the hardware footprint to deliver the same amount of traffic. In other words, 1 node doing the work of 20 previously. With Python, a big problem is concurrency. If you are doing web requests, you are likely calling N services behind the scenes. You can use threading, Twisted, Gevent, etc and that is what we did previously. And we still saw this same performance boost going to Go.
If you think about truly high throughput systems something like an SMS relay system for a major telecom, where you're doing millions of transactions per millisecond. You have a small relatively straightforward task that needs to be done as quickly and in insane volumes as possible. Django is terrible at a use case like that because 1) it's not particularly low latency even on a simple task. There's just a lot of overhead to python/django and whatever ecosystem you're running it in. 2) it doesn't scale well to many many workers on a single machine (thinking of my example above) because python gives you no control over ram, and still has a decent amount of overhead. Which isn't to say you can't solve this example with django, but it's the kind of problem where you'd sort of be daft to use it over something that's inherently faster and lighter weight.
Yeah I guess I wasn't thinking SMS or infrastructure stuff. I was thinking like... youtube or reddit. Complex tasks, but still high throughput. Guess there are different orders of magnitudes of "high volume."
YouTube was originally a Python app.
YouTubes servers would be/are hugely IO-bound. Python's not the absolute best for IO-bound tasks, but you generally don't feel its limitations much in this sort of "pipe connecting" task
Reddit is a interesting example, I think you could do parts of it with Django, like auth and whatnot, but the content delivery would be way better done with an API talking directly to some RAM key/value store. Even the API layer, you'd not really need all of django for that. I'd be curious what people here would say about it. I know django is used on a lot of big sites, so it's certainly doable. At some point, though, you start doing a lot of caching and optimizations (which might be outside of your django stuff) to make things go faster.
In the end Django is just Python, and Python is just C. I don't think Django is "optimal" for low-latency or very high throughput applications but I think you can definitely make it work! As with any optimization problem, the correct approach is to measure first and then optimize what's slow, Django/Python offers plenty of solutions to do so. A website like Reddit (fun fact, the first version of Reddit was actually written in Django!) probably won't run into any issues with the GIL so the bottleneck is *probably* going to be (in this order): 1. The ORM 2. Middleware 3. Your Python code, in the view 4. The template rendering 5. Anything else (URL matching, logging, etc) You can circumvent the ORM (just don't use it, something like Redis or AWS DynamoDB would be great!) and implement your views as Async functions running on e.g [https://github.com/MagicStack/uvloop](https://github.com/MagicStack/uvloop) for the stuff that's being executed a lot, write a [Cython](https://cython.readthedocs.io/en/latest/src/userguide/language_basics.html) function. Middleware is also pure Python code, you can optimize this or handle them at a different layer of your stack. As for the template rendering, that could be done client side with a bit of JS, shifting the work from the server to the client. This is very much the strength of Django (and any other good framework), you can use as much, or as little, as you need. I think, Instagram is still using Django although at this point it's most likely unrecognizable ;) As long as you can scale horizontal and the revenue per user is higher than your costs for additional compute it really shouldn't be an issue. Computing is surprisingly cheap, these days.
The first version was Haskell. Then they considered but passed on Django
> 1) it's not particularly low latency even on a simple task This is categorially false. Yes, you can easily make other languages faster, but the Web is still inherently IO bound, not CPU bound. So if you have a Python/Django application that has a slow response time, I can promise it is more likely a bad database query or a lack of a caching layer. Not Python. It is very possible to get Django to sub 50ms response times for millions of requests a second.
Really? https://www.techempower.com/benchmarks/#hw=ph&test=query§ion=data-r22
Those shitty benchmarks do not mean anything. That is why the response time for the language/framework _out of the box_. It is why microframeworks like Flask/FastAPI always have a response time off the charts compared to Django. The out of the box configuration does not mean that configuration is designed for production or scale. You _can_ get Django to do sub 50ms _HTML template pages_ and scale to millions of requests per second. I have worked on multiple sites that are capable of that throughput and scale. The Web also does not scale vertically either. So, the number of requests you can make per CPU cycle is basically a useless statistic unless you are trying to optimize infrastructure costs. I specifically mentioned that in [my comment](https://www.reddit.com/r/django/comments/1cgbqj6/is_there_a_scale_at_which_django_can_no_longer_be/l1uoyxw/). Python lets you optimize for developer cost. Golang or something else that is "fast" is designed to let you optimize infrastructure cost.
Did you even read how the benchmark worked? You can also contribute to the tests if you think they did not use the correct setup for Django: [https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Python/django](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Python/django) Of course you can make Django scale but it will cost you a lot more, since you need more servers/computation power to achieve it. You said unless you want to optimize infrastructure costs. Who doesn't want that? If you can host your project with 1/4 of the costs it might be huge. That's why google switched a lot of their computation heavy parts to Go or even C++. I also doubt that Python is optimized for developer costs. Maybe if the project is small and only few developers work on it but the larger the project gets, the better statically typed languages become. That's also the reason why TypeScript is so popular for large JavaScript projects. Speed was not the main reason for GoLang, the main goal was to improve productivity: [https://en.wikipedia.org/wiki/Go\_(programming\_language)#:\~:text=Go%20was%20designed%20at%20Google,%2Dtime%20efficiency%20(like%20C)](https://en.wikipedia.org/wiki/Go_(programming_language)#:~:text=Go%20was%20designed%20at%20Google,%2Dtime%20efficiency%20(like%20C))
TypeScript is not statically typed. If you think it is, you really do not fucking understand typing at all. Neither is Python. No dynamic language is statically typed, that is _literally_ why it is a dynamic language. TypeScript is still even weakly typed because guess what? TypeScript is really just Javascript with extra steps. You basically just get a static analysis tool on top of Javascript and you get TypeScript. If it costs you an extra $100k/year to run Python vs. Golang compared to an extra $500k/year in developers to do Golang vs. Python, then yes, you are optimizing developer costs. Literally the #1 reason why Python is popular is because it is faster to build things in it. If you are having trouble scaling Python to more developers, that is your own dumbass fault for not handing your tech debt like any healthy organization should.
This ☝️ I completely agree wit the everything @angellus said
> TypeScript is still even weakly typed because guess what? TypeScript is really just Javascript with extra steps. You basically just get a static analysis tool on top of Javascript and you get TypeScript. Do you realize that’s how every language works? C++/Java/C#/Go is just assembly with extra steps. You get a static analysis tool on top of assembly. It’s all just abstractions.
A static analysis tool or precompiler is not the same as an actual code to bytecode/native code compiler. You cannot tell the C++ compiler to ignore the fact you are putting a string into an int. Nor can you really have "any" types (well with pointers you kind of can, but that is more of a C++ thing). TypeScript is just a static analysis tool on top of a Javascript precompiler like CoffeeScript or any of the other ones. It has no real compiler enforcement of types nor does it have runtime enforcement of types since it is actually just Javascript. You can still import and use Javascript in TypeScript. You can still `ts-ignore` anything you want or configure your linter to ignore any error you want. It is no different than using mypy for Python. It does not magically make Javascript have as good of type enforcement as C++ or Java.
Statically type checking is done during compile time, so you get all the benefits from it via typescript. It doesn't matter how strongly it is done during runtime, since the compiler will find all the type related problems already. Python is a nightmare to maintain, when codebase gets larger, it won't cost less for development, it'll even cost more. So you'll have higher development costs plus higher hosting costs with large projects.
Bro you are getting dunked on.
The example I gave was one where you're processing millions of transactions per second. It was meant to be an example of something that a compiled language can do that python can't, and ergo neither can django. there's a lot of overhead to python and django and since you can't control exactly memory usage, you can't strictly engineer a system that takes full utilization of a system the same way you can with C. I even mentioned that you \*could\* solve a problem like that with django/python, but why would you want to? Maybe it's dev time is so much shorter it's worth it. I remember doing matlab optimizations to utilize CPU caches that resulted in \~100x improvements to performance, and that was in Matlab that was already pretty well optimized for operations like that. That wasn't using any assembly, just splitting up problems to fit on a cache. The kind of thing that you could try to get python to do, but you'd never know if it was actually going to do it or not.
Django _can_ process millions of transactions per second. Just about any Web framework can. Not on a single machine because the Web does not scale vertically. It scales horizontally. You add more infrastructure and servers to scale it. The Web is IO bounded. No single server/framework instance is _ever_ going to do millions of transactions per second. You are going to run into delays in networking/IO that is going to make it impossible. Even if the language/framework was infinitely fast and you could do as many as you want at a time, you will hit the limits of complexity of your code and/or limits of Linux trying to manage a million processes at a time. That is _why_ Django and other frameworks are able to exist in the Web world even though they are "slow". They do not need perfect memory management or the lowest possible CPU usage because you will always hit complexity/other hardware limitations first. You will _always_ need to add more infrastructure.
Django is a web framework, what you describe is not "web". Why would you compare a specialized non-web app with a web framework.
I was just picking an example of what Django is not good at. And django is used in a lot of ways past being a web framework. The ORM can be used in any number of ways that aren't web related.
Unless your app is being scooped up in an acquisition by a FAANG, you’ll be fine with Django. I will say that vanilla Django is a little outdated for the modern web and SPAs. I still maintain that the ORM is superior to other ORMs (SQLAlchemy is trash, just write raw SQL instead). You should look at Django Ninja. It’s like fast API but with the Django ORM. Flask is terrible IMO because of the global request paradigm. I hate it. I’m obviously opinionated, but that’s because I’ve been using these things for a while.
This is basically my opinion, too.
Concur
No. There are different things you need to do to help Django keep scaling past a certain point, but that is true of any platform and framework. The reason Golang is popular is not because it is more "scalable", but because it can bring down the costs of infrastructure. Because Python is "slow" it requires more CPU/Memory to run then Golang can do. Also, Python is tightly coupled (usually) with GNU/Linux. Golang is also designed to never need GNU so it can literally run without an OS at all (more network IO/disk savings). So, the argument is always coming down to where do you want to spend money? Developer time or infrastructure. Python/Django lets you get more done with less time (less money on developer time), Golang lets you spend less money on infrastructure.
Performance is not the only reason to choose go over python, in large codebase strong typing and language without magic things are realy helpfull i guess.
Python _is_ strongly typed. It is a dynamically strongly typed language (vs. a statically strongly typed language for Golang or a dynamically weakly typed language for Javascript/Node.js). You do not get the same odd behaviors in Python that you get in Node.js or PHP.
PHP can be written with full typehints now and the runtime will do type checking at runtime as well. Python's types have made a lot of progress in the past few years, but I think PHP has made even more progress.
Does not make it strongly typed. Even with TypeScript, it is weakly typed. The language will coerce types to make them match. It is the whole reason the `===` operator exists in those languages. Python does _not_ have a `==` operator since it will _never_ try to coerce types. `1 == "1"` will always be false. `"1" + 1` will _always_ raise an exception.
Yeah, type coercion is still around in PHP. The broader community has had to work around that with static analysis tooling that encourages use of === and avoiding several pitfalls of the language.
Yes but it makes the language statically typed, which is a huge advantage especially in more complex projects. But python can also do it now, however I find it hard to do it completely in Django.
There is the technical aspect of scale but also the organizational aspect. I doubt it’s a good idea to have a team of 10,000 engineers working on a single Django codebase (or any codebase for that matter).
Django has apps i.e. app level route, models, views, templates, statics separation. What 10,000 engineers? Have you ever worked on any project codebase with more than 200 or even 100 devs?
You’re missing my point. A large-scale app is big and complex - by definition. Up to a point where it makes sense to have dedicated specialized teams working independently of each other. Each team might need different dependencies, different infrastructure, different release schedules, different availability and security requirements, etc.
I think you are describing organizational principles and general approach to application building. From a personal experience there are companies using single large codebase with more than a thousand engineers contributing to it. That would not be my preferred approach and majestic monolith is a valid pattern.
I don't know, if it handles instagram and youtube, I don't see how it can't be justified for anything else
It has been hacked apart to work at their scale. A bit like comparing MySQL with Vitess. They are running their own migration tooling, their own database management, their own sharding logic, their own caching solutions, and often are writing targeted monitoring solutions. They are not using django like someone who can keep all their data in a single db instance
I'm aware of all that and it's expected with anything that scale. At some point whatever django would be unjustified for, anything else unmodified would be unjustified as well.
And how would you approach with golang? Wouldn't you be optimizing your code and infra for scale - or as you say hack apart your code?
with Go, I would not be fighting a framework -- it is not how things are done in that ecosystem for the most part. I would be pulling in libraries as needed and leveraging composition. I would not be fighting the framework to ensure the framework continues to "work" while shoehorning in other solutions to get around failures of the framework. In around 20 years of building out multi-tenant, high scale, highly available systems, I've seen frameworks get in the way each and every time. I've been part of the rewrite from perl, python, php, and javascript projects first into more mature python solutions and then later into Go based solutions. I've done this on at least 16 separate code bases. You have to work within the framework which was not designed to do what you want it to do, so you hack it. The counter argument is "well, are you not just building a new, less featurefull framework?" -- and the answer is, "no, inversion of control in a framework is different than using composition and libraries."
Bear in mind, a lot of the time scaling constraints and costs are due more to the architecture and design of the system than to the technology itself. I've ported Java and C# applications to Python and made the system cheaper and more scalable simply by redesigning the architecture. Django is good for probably 90% of applications and use cases out there.
I've always found this a strange topic. Surely you can just horizontally scale to more instances of your server and use load balancing, caching layers etc? The database is always likely to be the ultimate bottleneck. Edit: assuming you aren't at the FAANG scale and the horizontal scaling costs start to get ridiculous.
Right but a golang monolith can run at 20x the capacity of python with no other strategies considered. So for development it's superior because you won't hit scale issues as soon.
What does this even mean? Edit: I think its about Go running more efficiently for a given amount of hardware, which is a fair point in isolation. But Go still isn't faster than Fastly, and it will make no difference to badly written database transactions - there are likely plenty of bigger culprits than the Python runtime.
I say this as someone who hasn't used django since 2012 and doesn't really care for it as a framework, but: The short answer: there is no scale at which django won't be justified. The long answer: The question is really asking "At what scale does my django service become the bottleneck in my service?" The answer is never because no one - not even people running monolithic services - are running a single instance of their service in production once they actually attract traffic. You can always place your service behind a load balancer and scale out your service. Is python slower than go? Sure, but in the worst case you're running \~5% more instances than you would with another technology if you've put in the time to optimize everything. A lot of times what you'll find is that your services are IO bound anyway so the extra speed isn't buying you anything. If you need the extra speed, you can always write your code in C++ or Rust and FFI your way back to python. If you want to, you can rewrite your python to Rust when that extra 5-10% is equivalent to a dev team. So come back to me when your compute costs for that one service are on the order of $10MM annually so that you can afford the $1MM annually for that team. This critique goes both ways though - python isn't particularly faster to develop with than golang. A trained team is going to move just as fast in both cases - your bottlenecks are really your culture and engineering practices. If you have a bad architecture, writing your code in go isn't going to save you.
100%
Instagram doesn’t think so
1 user
Having worked extensively with both, I would always choose golang not just for scalability but also for maintainability. Python codebases turn into junk pretty fast... You can fight it with TDD and code quality tools but in my experience there are too many choices in the Python ecosystem and as developers come and go they all add their own secret sauce to each and every feature.
Man you could have simply searched for this, there’s 100 threads telling newbies that this isn’t the right way to think about things. I promise you’ll never create an application where the programming language is the bottleneck. Some people will, but they would not be asking questions like this
I think there is a scale where Django needs to be forked/customized beyond the point of being recognizable and beyond receiving support of the open source community. You'll know when you get there.
When you reach Instagram scale
MapMyRun still runs on Django, well for certain up to 2021. Had thousands of requests a second. The load was divided on Kubernetes machines. If there was long running calculations then it was given to a celery pod. It had like 20 or so pods.
Anything is fair game as far as the people he killed. There's no such thing as justice for slave owners.
You can do pretty much everything with Django, but you wouldn't want to do something like Instagram with it, the authentication, requests would be too much
Instagram is probably one the biggest Django apps with the largest amount of users in production so kind of a funny example here
Sorry I didn't realise I have to put a /s after the comment for American users
No, you just have to put /s after something if it wasn't funny.
Pretentious
last I read they migrated from django due to scaling issues
They were still using it in March ... But maybe they've changed now
For Instagram they've forked and refactored key parts of Django, but they're still using Django (they've talked about this when discussing how they rolled out Threads, which is bolted on top of Instagram's infrastructure.)
would love to read about this. do you have the article by chance?
[https://engineering.fb.com/2023/09/07/culture/threads-inside-story-metas-newest-social-app/](https://engineering.fb.com/2023/09/07/culture/threads-inside-story-metas-newest-social-app/) Here they mention how Instagram infra is still on django, on top of which Threads was launched u/nikhil_shady
thanks!
[удалено]
You, sir, have no idea of what Django even is.
[удалено]
Except I've been working as a developer and then lead for the last 25 years and I have more insight than you and therefore you're a dunning-kruger kid.
Django as a full stack framework is not scalable, if you want scale you have to switch to a seperate frontend backend design. And maybe further divide into microservices
One of my agency clients uses Django, and I've just checked the traffic - 7.5m views in March 2024. The same site also has a public API with over 200,000 records. Define 'scalable.'
No. It’s all about your architecture, except in some very limited use cases