Tag Archives: google

Golang contribution “journey”

Golang has probably the worst contribution model among every other project I worked. Even linux kernel is kawai compared to this.

They have finally merged my wine time fix into runtime/windows subsystem, which took 3 month of negotiations and required 2 almost full rewrite and more importantly, Google’s code contribution policy.

Wine does not support memory mapped update of the timer structures – it is similar to linux vsyscall (or linux copied it from windows), i.e. kernel timer periodically updates some fixed address which can be mapped into userspace and read without using syscall mechanism which is noticeably slower. Wine does not have it (and frankly it can not, since it runs in userspace), but the whole golang windows port is based on this windows feature. I implemented a fallback mechanism which uses plain windows syscall to get the time and QPC counters to implement monotonic time.

But this post is not about technical details of the time subsystem in golang, but instead about stupidity of the google’s contribution policy.

Google contribution policy requires you to have a google account, which it may not create for whatever obscure reason, it just fails and nothing moves forward. No oauth as an industry standard, but only ugly google’s own account. I wonder whether it will force me to use google+ to login.

If you ask another person to contribute into golang on your behalf, they do not accept your patch, since they do not know whether you signed or not contribution agreement (event though the person who sends the patch and authors it does sign this shit). Google forbids to discuss and review patch on github, you can only create an issue with complain, they force to use Gerrit and follow golang’s own horrible contribution steps based on single-commit approach in git.

Technical management at google (and frankly many other places) breaks the whole idea of the fun behind working with open source project.

Please be not like google.

Mesa – Google’s analytic data warehousing system

I will bring back various tech article analysis in this blog

Let’s start with Mesa – Google’s adwords storage and query system. Albeit the article is very self-advertising and a little bit pathos, it still shows some very interesting details on how to design systems which should scale to trillions operations a day.

In particular, data is versioned and never overwritten. Old versions are immutable. Queries aka transactions operate on given version, which are asynchronously propagated over the google’s world datacenters. The only sync part is versioning database – committer updates/pins new version after update transaction criteria has been met, like number of datacenters have a copy and so on. As usual – it is Paxos. Looks like everything in Google uses Paxos, not Raft. Recovery, data movement and garbage collection tasks are also performed asynchronously. Since the only sync operation is version update, committer (updating client) can write heavy data in parallel into the storage and only then ‘attach’ version to those chunks.

Data is stored in (table, key, value) format, and keys are split in chunks. Data is stored in compressed columnar format within each chunk. Well, Mesa uses LevelDB, it explains and simplifies such things.

Mesa uses very interesting versioning – besides so called singletons – single version update, there are also ranges like [0, v0], [v1, v5], [v6, vN] named ‘deltas’ which are basically batches of key updates with appropriate versions. There are async processes which updates version ranges: compaction [0, vX] and new batch creation processes. For example to operate a query on version 7 in example above one could ‘join’ versions [0, v0], [v1, v5] and apply singletons v6 and v7. Async processes calculate various ‘common’ ranges of versions to speed up querying.

Because of versioning Mesa doesn’t need locks between query and update processing – both can be run heavily in parallel.

Here is an article: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42851.pdf