SECL group

+1 (929) 237-12-11
02.18.2016

High-load Websites and Applications in Python / Django

Our company makes a development of the big web applications on Python / Django for a quite long time. For ease of comparison sites between themselves, we give some statistics for each project. In general, there are many websites in the world, which are written on Python / Django, so we will cover the most interesting of them on our opinion.

A small table of well-known projects(based on the data http://builtwith.com/):

Website name Link Language CMS/Framework TYC/PR Server Hosting Providers Analytics and Tracking JavaScript Libraries Widgets Other Media Mapping
Pinterest https://www.pinterest.com/ Python Django 5600/9 nginx Amazon Google Analytics, Facebook Domain Insights, Optimizely Facebook, jQuery, Backbone.js, Modernizr Pinterest, Facebok Like, Facebook Facepile, Imgur Javascript, CSS, Iframe Vimeo
Disqus http://disqus.com/ Python Django 4000/9 nginx Softlayer New Relic, Google Analytics, comScore, Mixpanel Sentry, Underscore.js, jQuery, Modernizr, selectivizr, RequireJS Wufoo, Typography, Disqus Javascript, CSS, Iframe, HTML5
Instagram http://instagram.com/ Python Django 12000/9 nginx Amazon Compuware Analytics, Google Analytics, Facebook Domain Insights ECMAScript 5 Shim, HTML 5 Shiv, jQuery Facebook Like Javascript, CSS, HTML5
Mahalo http://www.mahalo.com/ Python Django 120/6 nginx Facebook Domain Insights, Google Analytics, Quantcast Measurement Facebook SDK, Backbone.js, jQuery Google Plus One Platform Javascript, CSS, HTML5
Bitbucket https://bitbucket.org/ Python Django 850/7 nginx NTT America New Relic, Google Analytics Underscore.js, jQuery, Mustache, Backbone.js, RequireJS reCAPTCHA, Typekit Javascript, CSS, Iframe, HTML5
Lanyrd http://lanyrd.com/ Python Django 50/6 nginx Softlayer Facebook Domain Insights, Google Analytics, KISSmetrics LinkedIn Platform API, Twitter Platform, Facebook SDK, WebFont Loader, Mustache, jQuery Facebook Like, Google Plus One Platform, Twitter Tweet Button, LinkedIn Share Plugin, Fontdeck Iframe, Javascript, CSS, Meta Keywords, Facebook Page Administration, HTML5 jPlayer MapBox, Google Maps
Mozilla http://www.mozilla.org/, https://support.mozilla.org Python Django 9000/9 Apache Optimizely, Google Analytics Modernizr, PNG Fix, jQuery, matchMedia Facebook Like/Sharer, Google Custom Search Twitter Cards, Meta Description, CSS, Javascript
The Onion http://www.theonion.com/ Python Django 375/7 Microsoft, Personal, Web Server Linode Facebook Domain Insights, Google Analytics, Quantcast Measurement, Safe Count, Dynamic Logic, comScore, Optimizely Twitter Platform, jQuery, Facebook SDK, Backbone.js Twitter Timeline, BuzzFeed, Typekit, Facebook Like Box, Press Plus Mobile Optimized, Meta Description, iFrame, HTML5, CSS, Javascript
Everyblock http://www.everyblock.com/ Python Django 35/7 nginx Amazon Facebook Domain Insights, Google Analytics jQuery Cookie, jQuery Google Font API, Webtype, Facebook Like, Tweeter Tweet Button Javascript, CSS, Iframe
Giant Bomb http://www.giantbomb.com/ Python Django 120/6 nginx Amazon New Relic, Nielsen//NetRatings, comScore, Google Analytics, Omniture SiteCatalyst jQuery/UI, SWFObject, Twitter Platform, Modernizr, Backbone.js Typekit, Facebook Like, Tweeter Tweet Button, ZeroClipboard Javascript, CSS, Iframe, HTML5
ComicVine http://www.comicvine.com/ Python Django 70/6 nginx Amazon New Relic, Nielsen//NetRatings, comScore, Google Analytics, Omniture SiteCatalyst jQuery/UI, SWFObject, Twitter Platform, Modernizr, Backbone.js TRUSTe, Facebook Like, Outbrain Content Rating Javascript, CSS, Iframe, HTML5
Michael Moore http://michaelmoore.com/ Python Django 110/6 nginx Amazon Quantcast Measurement, Google Analytics Twitter Platform, jQuery/UI, BBC Glow, SWFObject Twitter Timeline, Thickbox, AddThis XHTML, CSS, Iframe, Javascript YouTube, JW Player
National Geographic http://www.nationalgeographic.com/ Python - 2100/8 nginx, Apache Internap, Akami Hosted Core Audience, CrazyEgg, Google Analytics, New Relic, Nielsen//NetRatings, Facebook Domain Insights, SkyGlue SWFObject, RequireJS, jQuery/Cookie, Twitter Platform, Facebook SDK, Backbone.js Facebook Like/Sharer, AddThis, Twitter Tweet Button, LiveFyre Google Chrome IE Frame, CSS, HTML5, Javascript, XHTML, Pinterest No Pin, Twitter Cards Vimeo, jPlayer
Discovery http://www.discovery.com/ Python Django 0/8 Apache - Facebook Domain Insights, Nielsen//NetRatings, Dynamic Logic, Google eCommerce Conversion Tracking jQuery Smart App Banner, Facebook Like, Google Tag Manager Facebook Page Administration, Twitter Cards, HTML5, Iframe, CSS, Javascript - -
Dropbox https://www.dropbox.com/ Python - 4900/8 nginx - Facebook Domain Insights, Google Analytics jQuery, Lightbox, Facebook, Modernizr, SWFObject, Prototype Facebook Like Javascript, CSS, Iframe, HTML5, XHTML - -
Google https://www.google.com Python - - - - - - - Javascript, X-Frame-Options, HTML5 - Google Maps
The Guardian http://www.theguardian.com/ Python - 5800/8 nginx, Apache - MediaMath, Facebook Domain Insights, Google Analytics, Omniture SiteCatalyst, Buysight, Quantcast Measurement Twitter Platform, jQuery, Google API, SWFObject, Underscore.js, RequireJS Pinterest, Google Plus One Button, Twitter Tweet Button, LinkedIn Share Plugin, Facebook Like HTML5, XHTML Iframe, Twitter Cards, Javascript, CSS, Google Chrome IE Frame JW Playertd> -
The Washington Post http://www.washingtonpost.com/ Python, PHP - 3500/8 nginx, Apache nLayer Communications, Akami Hosted Google Analytics, Moat, Krux Digital, New Relic, Omniture SiteCatalyst Underscore.js, jQuery, Prototype, Twitter Platform, Yahoo User Interface, Facebook SDK, JSON Pinterest, Twitter Timeline, Google Font API, AddThis, Facebook Sharer, Facebook Like,Google Plus One Platform Facebook Page Administration, HTML5, Iframe, CSS, Javascript JW Player, Tube Mogul, Ooyala -
The New York Times http://www.nytimes.com/ Python - 6300/9 Apache, Sun ONE Web Server - Google Analytics, Moat, Krux Digital, OpinionLab Facebook SDK, Twitter Platform, jQuery, Facebook, Underscore.js, Backbone.js, Mustache Twitter Timeline, Facebook Like Box, Typekit, NextGEN Gallery Javascript, CSS, Iframe, HTML5, XHTML Livestream, Brightcove -
Yelp http://www.yelp.com/ Python Django 500/7 Apache - Facebook Domain Insights, Google Analytics - Facebook SDK, jQuery, Yahoo User Interface, LABjs Javascript, CSS, Iframe, HTML5 Ooyala Google Maps
Reddit http://www.reddit.com/ Python - 3500/8 - Amazon, Akamai Hosted Google Analytics Backbone.js, jQuery Imgur Javascript, CSS, Iframe, HTML5, XHTML - -
Facebook https://www.facebook.com/ Python, PHP - 170000/9 - - - - Facebook Like CSS, Javascript, HTML5 - -
Rdio http://www.rdio.com/ Python Django 80/8 nginx Level 3 Communications Optimizely, Google Analytics, Microsoft Adcenter, Omniture SiteCatalyst, New Relic LESS, Twitter Platform, Facebook SDK, jQuery, Modernizr, Backbone.js Twitter Follow Button, Facebook Like Button, Google Font API, Typekit Javascript, CSS, Iframe, HTML5 - -
Globo http://www.globo.com/ Python Django 700/7 Varnish - Google Analytics, comScore jQuery Facebook Like HTML5, Javascript, Iframe, CSS - -
Pitchfork http://pitchfork.com/ Python Django 325/7 Varnish, nginx Amazon New Relic, Google Analytics, comScore, Quantcast Measurement, Chartbeat Head JS, Facebook SDK, Google JS Api, Twitter Platform, jQuery Typekit CSS, Javascript, HTML5 - -
Politifact http://www.politifact.com/ Python Django 80/8 Varnish, nginx Amazon Google Analytics, Krux Digital, comScore, Facebook Domain Insights, Moat RequireJS, Facebook, jQuery, SWFObject, Backbone.js Lijit Widget, TRUSTe, Facebook Like Iframe, Facebook Page Administration, CSS, Javascript, XHTML - -
Foursquare https://foursquare.com/ Scala, Python - 2700/9 nginx Amazon Google Analytics Facebook, jQuery, IE Pinning - HTML5, CSS, Javascript - -
Quora http://www.quora.com/ Python - 325/7 - Amazon Google Analytics, Facebook Domain Insights Facebook SDK, jQuery Facebook Facepile/Like Javascript, CSS - -

Pinterest (social network)

Technologies that have been used:

Python as a programming language and as a framework has been chosen Django. Hosting - Amazon. MySQL is taken as the main database management system, caching of objects does memcached and collections of objects - Redis. Solr is used as a search platform, and Hadoop - for realization search and context mechanisms, analysis of data.

A little bit of statistics:

According to latest figures, working staff counts more than 140 employees. About 11 million unique visitors per week in December, 2011. 500 virtual machines in EC2, 410 terabytes of user data or 80 million different objects are stored in Amazon S3. Pinterest has about 70 million users according to French news agency Semiocast. Total, Pinterest becomes the fourth most popular social network in the United States after Facebook, Twitter and LinkedIn.

(http://en.wikipedia.org/wiki/Pinterest)

(original http://www.businessinsider.com/how-we-scaled-pinterest-2013-4?op=1, http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html)

Disqus (service)

Technologies that have been used:

Python as a programming language and as a framework has been chosen Django. Operating System - Linux. MySQL is taken as the main database management system, caching of objects does memcached as in the case with Pinterest. Over the load balancing is responsible HAProxy, for data replication - Slony.

A little bit of statistics:

33 people works in staff. The number of registered users is growing and currently amounts 50 million people. Site receives about 17,000 requests per second. More than 144 million unique visits per month only from the territory of the United States. Around 750,000 websites and blogs.

(source http://www.insight-it.ru/masshtabiruemost/arkhitektura-disqus/, original http://highscalability.com/blog/2010/10/26/scaling-disqus-to-75-million-comments-and-17000-rps.html, http://en.wikipedia.org/wiki/Disqus)

Instagram (photo and video application)

Technologies that have been used:

Ubuntu Linux 11.04 has been taken as the primary operating system. Python as a programming language and as a framework has been chosen Django. PostgreSQL has been taken as the main database management system, caching of objects does memcached, Redis acts as an additional data storage. Over the load balancing is responsible HAProxy. The project uses the infrastructure of Amazon, particularly EC2, ELB, Route, S3 and CloudFront. Solr has been taken as a search platform. Gearman is used to work with the tasks.

A little bit of statistics:

It all started with one small and weak server and two developers. On the first day about 25,000 users was registered. Today Instagram used by more than 200 million people and about 7 million users daily. About 20 billion photos have been published and 60 million are published daily. Facebook bought the service for 1 billion dollars in April, 2012.

(source http://expandedramblings.com/index.php/important-instagram-stats/, http://en.wikipedia.org/wiki/Instagram)

Reddit (news website)

Technologies that have been used:

Python as a programming language and as a framework has been chosen Pylons. PostgreSQL is taken as the main database management system, caching of objects does memcached. RabbitMQ is used for autonomic data processing. Over the load balancing is responsible HAProxy. Amazon CloudSearch is used as a search platform.

(source http://en.wikipedia.org/wiki/Reddit)

A little bit of statistics:

Approximately 112 million unique visits per month. 5,46 billion page views per month. 2,89 million people use Reddit. And all of this is served by the team of 28 people.

(source http://expandedramblings.com/index.php/reddit-stats/)

Dropbox (cloud storage)

Technologies that have been used:

Dropbox server and client application have been written on Python. The client uses development tools GUI, such as wxWidgets and Cocoa and other important Python libraries such as Twisted, ctypes and pywin32. Dropbox depends on librsync library, which is written on C. Information about the files is contained in the storage, which was realized on the basis of MySQL. Amazon S3 is used to store the files.

(source http://en.wikipedia.org/wiki/Dropbox_(service))

A little bit of statistics:

Staff consists of 110 employees. More than 50 million registered users. Every 3 minutes more than a million files are stored using the service. 500 million files are saved daily.

(source https://www.dropbox.com/static/docs/DropboxFactSheet.pdf)

Pitchfork (music e-zine)

Technologies that have been used:

Python as a programming language and as a framework has been chosen Django. Hosting - Amazon. MySQL is taken as the main database management system. In addition, is used PostgreSQL. Redis respond for caching of objects. ElasticSearch and Solr are used as a search platform. Also are used the services of Amazon (EC2, RDS, SES).

(source http://www.siteclass.com/www/pitchfork.com)

A little bit of statistics:

The project has more than 14.5 million visits per month (580,000 per day) of which 5.5 million are unique (410,000 per day). View of pages is 38.5 million per month (1.6 million per day). The number of registered users is 4.3 million on March-April, 2014.

(source https://www.quantcast.com/pitchfork.com)

Lanyrd (portal)

Technologies that have been used:

Python as a programming language and as a framework has been chosen Django. PostgreSQL is taken as the main database management system, caching of objects does memcached. Redis is used in combination with Celery to store intermediate results of tasks, which performs Celery asynchronously. Over the load balancing is responsible HAProxy. The project uses the infrastructure of Amazon, particularly S3. Solr is used as a search platform.

(source http://www.slideshare.net/InfoQ/inside-lanyrds-architecture)

A little bit of statistics:

The development team consists of only 6 people. That's what Andrew Godwin said in his presentation. About 900,000 unique visitors per month in March, 2014. The maximum value was in October, 2013 and amounted more than a million visitors (http://www.trafficestimate.com/lanyrd.com). Daily view on pages is about 55,000.

(source http://www.slideshare.net/InfoQ/inside-lanyrds-architecture)

Mozilla (software)

Technologies that have been used:

Mozilla company uses various programming languages, including Python, which is used widely: from writing build scripts to the website of company, Webmaker and other components, as well as synchronization server - minimalistic WSGI application using Paste to deploy on hosting and Sqlite3 as a database.

A little bit of statistics:

Today Mozilla projects use more than 500 million people. This is very big indicator, considering the competition. More than a thousand developers are involved in Mozilla projects.

(source http://expandedramblings.com/index.php/internet-browser-stats/)

Yelp (portal with elements of social network)

Technologies that have been used:

Ubuntu Linux has been taken as the primary operating system. Python as a programming language and as a framework has been chosen Django. MySQL is taken as the main database management system. Also Yelp uses services of Amazon particularly S3 to store logs and photos, and EMR. Solr/Lucene are used as a search platform. Over the load balancing are responsible HAProxy and LVS.

(source http://engineeringblog.yelp.com/, http://aws.amazon.com/solutions/case-studies/yelp/)

A little bit of statistics:

Site traffic is about 200 million people per month of which about 120 million are unique visits.(https://www.quantcast.com/yelp.com). During all work time of the project has been written more than 53 million user reviews worldwide.

(source http://expandedramblings.com/index.php/yelp-statistics/)

Foursquare (social network with the function of geopositioning)

Technologies that have been used:

The project is written on several languages, including Python, which is used to automate operational tasks and other processes. CentOS Linux is as the primary operating system. HAProxy is responsible over the load balancing and API-requests. MongoDB is taken as the main database management system, caching of objects does memcached. Part of information, namely photos of user, are stored in Amazon S3. Hadoop deals with the analysis. Solr and Elasticsearch has been taken as a search platform. Library s2 from Google in combination with PostGIS is used to search for geo-index. Kestrel responsibles for handling asynchronous tasks.

(source https://foursquare.com/about)

A little bit of statistics:

Staff consists of about 140 employees. About 40 thousand developers involved the project according to information over 2013. About 45 million users. The total number of check-ins are 5 billion worldwide and about 3 million are added every day.

Rdio (music service)

Technologies that have been used:

Rdio uses several programming languages. Particularly, part of the back-end is written on Python and as a framework has been chosen Django. MongoDB and MySQL are used to store information. Redis was chosen as an alternative of memcached.

A little bit of statistics:

Catalog of more than 20 million songs. Daily around 200,000 views of pages. About 300 thousand people use the services of the project during a month in the United States.

Google (search engine)

Technologies that have been used:

Google uses a variety of programming languages. Creator of Python has worked for a while in Google, so it's easy to assume that there is used this programming language. And so it is truth. Part of YouTube and search engine is written on Python, as well as many other components. Briefly, LevelDB is used as the main database management system. Closure is also used to work with JavaScript. It is a kind of a set of tools created by developers of Google.

Technologies that have been used:

Statistics of Google is known everyone and everywhere, but let's look some indicators. Monthly through the search passes 12,477 billion queries. Google takes about 67% of the search market of the United States. 191 million of people per month are unique visits. Staff of the company consists of 53 891 people.

(source http://expandedramblings.com/index.php/by-the-numbers-a-gigantic-list-of-google-stats-and-facts/)

Quora (social service "Questions and Answers")

Technologies that have been used:

Python as a programming language. Hosting - Amazon. MySQL is taken as the main database management system, caching of objects does memcached. Over the load balancing is responsible HAProxy.

A little bit of statistics:

Staff consists of 72 employees. The number of unique visits is more than one million according to the information of February, 2014. Monthly 1,126,000 people use service Quora (data for 2013).

(source http://www.quora.com/How-much-traffic-does-Quora-get, http://techcrunch.com/2013/11/12/quora-confirms-its-favoring-search-ads-for-eventual-monetization-launches-author-stats-tool/)

Summarize

As you can see, a lot of famous projects use Python / Django and that is justified. Python is very interesting modern programming language, which is rapidly gaining popularity now. At the moment, we observe a big demand for programming on Python / Django, because it's quality! Demand for projects gives rise of demand for specialists.

If you want to develop a high load project, we advise you to consider Python / Django as a technical platform. Well, and something for colleagues. If you want to be a programmer or learn a new programming language - Python is one of the best options.

P.S. To get our new articles before the rest or not to miss new publications – subscribe to us on Facebook, Twitter, and LinkedIn

About the Author

Andy Astafyev
Andy Astafyev
Middle Project Manager, SECL Group Corporation
+1 (929) 237–12–11 info@seclgroup.com

UP
Request Form