kraken.io - our powerful servers

Technology Overview · Kraken.io

Kraken.io Spined HTML

Technology Overview · Kraken.ioWeb InterfaceFeaturesPluginsAPI DocsSupportMy Account Pricing & SignupGet Started FREELog InSign UpAbout KrakenWeb InterfacePluginsFeaturesAPI DocsSupportKraken BlogTechnology OverviewHide Table of ContentsShow Table of ContentsGeneral InformationAbout Kraken.ioTechnologyMedia KitKraken.io in PressContact UsLegal InformationTerms of ServicePrivacy PolicyData Processing AddendumImpressumIntroductionThis section will touch upon how Kraken.io built and scaled an image optimization platform which serves millions of requests per day, with the goal of maintaining upper performance at all times while keeping financing as low as possible. We present our infrastructure as it is in its current state at the time of writing, and touch upon some of the interesting things we learned in order to get it here.API and Webhook Delivery MachinesHardwareImage optimization and recompression has enormous processing requirements. Cloud was never an option for us as we are continuously trying to lower our total forfeit of ownership.All single-socket machines (API, Web, Load Balancers, Webhook Delivery) are currently running Xeon E3-1280 v5 (Skylake). For Optimization Cluster where all the nonflexible work is washed-up we use 2 x Xeon E5-2697 v3 per machine with 128 GB RAM and four SSD nonflexible drives in RAID-1 setup for mirroring. With HT enabled the whilom setup gives us wangle to 28 physical cores and 56 threads per Cluster machine.Kraken.io's platform is both CPU and I/O intensive, performing heavy processing on a large number of files. To proceeds increasingly performance on the I/O level we will be rolling out PCIe-SSD drives for our API, Cluster and Storage machines.One of our optimization workers (2 x Intel Xeon E5-2697)Data StoreWe use MongoDB in a Replica setup on three self-sustaining machines as our primary data store. As our dataset is relatively small and we use capped collections for all time series data DB sharding was something we never really have considered.The second data store is Redis in a Sentinel setup (for the same reasons as the above). Used primarily as a message usurer for the task queue and session management on Kraken.io frontend.Provisioning, Discovery and Software DeploymentsEvery new machine we install is managed and configured by Foreman. We alimony all the configuration in Puppet so bringing a new machine to production-ready state only takes a couple of clicks.Software deployment is washed-up through Capistrano. We use similar recipes for scrutinizingly all our applications as all applications are written in Node. Integration with Slack is very helpful when we need to pinpoint a specific deployment that happened in the past and correlate that with the data misogynist in ServerDensity or ElasticSearch.File StorageIn a previous generation of Kraken.io we used to store optimized resources directly on the same machines that did the optimisation work. As we've decoupled the roles (API, Web, Processing Cluster and Storage) we've found ourselves in the firsthand need of a scalable network file system. GlusterFS was easy to setup and it is easy to maintain.We have millions of images flying over the wire from using servers to GlusterFS machines. It is very important for us not to move those files too often. Once saved in Gluster an image stays there until its will-less removal.Load BalancersBoth external and internal load balancers are Nginx-based with Keepalived on each one of them. Even if both of our externals go lanugo the internal ones will automatically promote themselves and moreover serve the public traffic.We don't use any HTTP servers on our internal machines. All the internal traffic is reverse-proxied from load balancers directly to Node applications.NetworkingAs we're moreover redundant on the uplink level (two self-sustaining 10 Gbps uplinks) we needed at least two switches per rack and two Ethernet controllers per machine. As the racks grow and each machine occupies five ports on the switch (BMC, Uplink A and B to Controller 1 and Uplink A and B to Controller 2) currently we're running four HP ProCurve switches per rack.Monitoring and AlertingIn the previous generation of Kraken.io we used Sensu, Graphite and InfluxDB. As we wanted to shift our full sustentation to the product itself and not maintain and monitor the monitoring tools we needed a SaaS that would take that pain away. After testing several services we finally settled with ServerDensity as our primary monitoring and alerting tool for all our machines and it works flawlessly so far.ServerDensity metrics displayed in our officeData MiningAs we try to alimony the number of supported technologies to a yellowish minimum we use an external ElasticSearch provider. On an stereotype day we ship 2GB of logs for remoter processing and data mining.As we're constantly working on improving the optimization stack we need to be worldly-wise to immediately track the results of our deployments. At peak loads it is unbearable to do a small tweak in the Optimization Cluster and get meaningful data in a couple of minutes.ProductFeaturesPricing & SignupFree Web InterfaceSystem StatusCompanyAbout Kraken.ioMedia KitTechnologyContact UsResourcesAPI DocumentationSupport CenterWordPress PluginMagento ModuleConnectBlogTwitterGitHubYouTubeCopyright © 2013 — 2018 Nekkra UGImpressum | Terms of Service | Privacy Policy