Saturday, January 21, 2017

Nicely put

John Scalzi takes a moment, and a deep breath, for a considered exploration of some famous words spoken by Dr. King, but actually written 150 years ago by Theodore Parker, in a marvelous essay: The New Year and the Bend of the Arc

We are human. We are not perfect. We will not arrive at a perfect justice, any more than we will achieve a perfect union. But just as we work toward a more perfect union, so too we bend the arc toward justice, knowing the closer we get, the better we and our lives are, as individuals, as communities, as a nation and as a world. This is a life’s work, not just work for a moment, or day, or year. You won’t see the final result. There isn’t one. It doesn’t mean the work doesn’t matter. It matters.

The January deluge continues

Check THIS out:

Shasta and Oroville BOTH at 125% of historical average!

Don Pedro is slamming too, already at 90% of capacity (and it's still mid-January!!).

Trinity and Folsom are looking fine, and even New Melones, scraping the bottom just 4 months ago, is up to two thirds of its historical average (we get our East Bay Municipal Utility District supplies from New Melones, btw).

Meanwhile, the 36-hour forecast is for 2-4 FEET of additional snow in the core region of the central Sierras (roughly, the headwaters basin of the Kings River).

The main weather concern now turns to yet another incoming storm which has the potential to be the strongest storm to impact our area so far this season. WV imagery is indicating a moist baroclinic band stretching from 130w out to 170w which is taking aim at central California. GOES high density winds are indicating that this feature is being pulled up by a strong upper low centered near 47n/143w and being pushed eastward straight toward central California. This deep moisture fetch is progged to reach our area by this evening and continue through at least midday Monday. During this time frame periods of moderate to heavy precipitation are expected across our area along with increased winds in favored areas. As the result of the anticipated heavy snowfall and strong winds in the mountains, a Winter Storm Warning has been issued for the southern Sierra Nevada where 2 to 4 feet of additional snowfall are expected over the higher elevations. Snow levels will be between 4500 and 5000 feet on satruday evening when this system arrives and rise to 7500 to 8000 feet by Sunday afternoon in a modified tropcial airmass. As the large offshore upper low approaches the norcal coast Sunday night, cooler air will push into California and snow levels will lower to 3500 to 4000 feet by Monday. Over the Kern County mountains a strong pressure gradient will result in very strong winds on Sunday with gusts up to 90 mph possible in the Grapevine area. The precipitation over the passes will turn to snow on Sunday night as the cooler air pushes in.

Please be careful if you're one of those people trying to drive the Grapevine this weekend. When the road freezes and the wind blows, it's one of the more dangerous roads in the country.

But I bet the Shasta pack is loving this.

Rain, rain, snow, rain. Oh, how sweet it is.

Friday, January 20, 2017

This and that

My challenge is simple: to master the nuances of my new job, all I have to do is to be comfortable with every page of this amazing 500 page masterwork: Is Parallel Programming Hard, And, If So, What Can You Do About It?. Happily, McKenney is a wonderful writer, but where will I find the time?

In the meantime, there's also a few other things that crept in while I was sleeping:

  • Who is Anna-Senpai, the Mirai Worm Author?
    The story you’re reading now is the result of hundreds of hours of research. At times, I was desperately seeking the missing link between seemingly unrelated people and events; sometimes I was inundated with huge amounts of information — much of it intentionally false or misleading — and left to search for kernels of truth hidden among the dross. If you’ve ever wondered why it seems that so few Internet criminals are brought to justice, I can tell you that the sheer amount of persistence and investigative resources required to piece together who’s done what to whom (and why) in the online era is tremendous.
  • Cloudflare’s Transparency Report for Second Half 2016 and an Additional Disclosure for 2013
    Because of the gag order, I had to sit in silence, implicitly confirming the point in the mind of the staffer. At the time, I knew for a certainty that the FBI’s interpretation of the statute diverged from hers (and presumably that of her boss).

    Cloudflare fought this battle for four years even after the request for customer information had been dismissed. In addition to protecting our customers’ information, we want to remain a vigorous participation in public policy discussions about our services and public law enforcement efforts. The gag rule did not allow that.

    Now that this gag order has been lifted, Cloudflare is able to publish a more accurate transparency report to its customers and constituents. For us, this is not the end of the story, but the beginning of a more robust, fact-informed debate.

  • Operation Grand Mars: a comprehensive profile of Carbanak activity in 2016/17
    Trustwave has tracked Carbanak activity in the latter half of 2016 and found them to be targeting hospitality and retail victims in Europe and North America, specifically targeting their internal corporate secrets and protected payment card data. Trustwave published a blog earlier in the year within initial findings but has now released the complete malicious campaign profile in a 45-page Advanced Threat Report. The blog post below is a summary of the malicious profile we have developed for this actor. The complete report on this activity can be found here:
  • Babai Strikes Back
    We were watching history. From the talk I tweeted the new news though Bill Cook, also in the audience, beat me to the punch. Babai went on to describe the issue, an error in the analysis of the running time in the recursion, and the fix, basically a way to avoid that recursive step, but I can't do it justice here. At the end he proclaimed "So it is quasipolynomial time again". And so it was.
  • On exp(exp(sqrt(log n))) algorithms.
    Improving quantitative parameters such as running time or approximation factor is very useful as intermediate challenge problems that force us to create new ideas, but ultimately the important contribution of a theoretical work is the ideas it introduces and not the actual numbers.
  • No One Questioned This Hedge Fund’s Madoff-Like Returns
    But until Murray Huberfeld, who founded Platinum with Nordlicht, was caught up in a New York City municipal-corruption probe in June, no one at the fund had been charged with wrongdoing. Within weeks of Huberfeld’s arrest, federal agents raided Platinum’s midtown Manhattan office. On Dec. 19, Nordlicht and six others were arrested in what the government called a $1 billion fraud. Nordlicht and Huberfeld have pleaded not guilty, and Platinum’s main fund is being wound down after filing for bankruptcy. Montieth Illingworth, a spokesman for Platinum, declined to comment.
  • Why Are So Many Bee Trucks Tipping Over?
    All of this is to say that I have little more insight, except that you definitely shouldn’t trust anyone! Except for maybe bees, considering they continue to provide us avocados and almonds and blueberries and all the foods that we’ve pushed to the edge of extinction, including the bees themselves, because we’re reckless as hell.
  • China’s WTO Entry, 15 Years On
    The state’s hand was clear, but not in ways that were obviously forbidden by the WTO. Or at least not in ways that have been successfully challenged in the WTO. Firms’ investment decisions aren’t technically government procurement if the investment is for the provision of a commercial service, and the state’s guidance isn’t always written down. Yet even today the preferences provided for local firms in strategic sectors, like medical equipment, aren’t exactly a secret that China tries all that hard to hide.


    The initial China shock overlaps with the dollar shock. WTO accession made producing in China for the global market attractive, but did not made China into a great market for manufacturers looking to sell globally produced goods to China. Successful WTO challenges to individual Chinese practices haven’t changed the overall pattern—China’s imports of manufactures for its own use have slid steadily relative to China’s GDP after WTO accession. The more-limited-than-expected gains for manufacturers looking to sell to China though didn’t lead (until now, when the China shock is arguably starting to fade) to a serious reconsideration of the basic gains from China’s asymmetric integration, in part because U.S. and European firms captured many of the initial gains of China’s export success. And some “within the rules” remedies weren’t used as aggressively as they could have been to challenge China’s currency management and other discriminatory practices during the years immediately after China joined the WTO.

  • Cooperation at the Tigris‎: U.S. and Iraqi Efforts to Maintain the Mosul Dam
    U.S. and Iraqi scientists estimate that a breach of the Mosul Dam, one of the largest pieces of water infrastructure in the Middle East, could have a devastating impact for over a million Iraqis along the Tigris River, from Mosul itself all the way down to Baghdad. Its failure could also be a threat to U.S. personnel and our Coalition members working alongside Iraqis and drastically reshape the nature of the fight against ISIL in northern Iraq. Today, thanks to close U.S.-Iraqi partnership, however, Mosul Dam is on a path toward greater stability.
  • CS Responder Trans-Oceanic Cable Layer
    Laying fiber optic cables with repeaters along the ocean floor raises super-interesting technical challenges. I recently visited the CS Responder, a trans-ocean cable-laying ship
  • The greatest chess game ever played
    Garry Kasparov, who is one of the top chess players ever, said that his 1999 match against Veselin Topalov was the greatest game of chess he ever played. In this video, MatoJelic goes through the game, move by move.

Thursday, January 19, 2017

Up, up and away!

Monday I'll report to work, bright and early, to a new office, in Salesforce West, the 6th tallest building in San Francisco.

Although this won't be the first time I've worked in a high-rise, I've generally avoided such office settings over the years.

But that's getting hard to do, at least in San Francisco, where an explosion of new development surrounds the Transbay Terminal development area, just one block away from my new office building.

A lot of this burst of development is due to changes to the zoning laws around downtown San Francisco, as well as considerable improvements in construction techniques.

Already built and operational are:

But that's just the beginning, for here come:

And, of course, the Big Daddy of them all, the Salesforce Tower, where the concrete was poured for the top floor just last week.

The new tower is just across the street, so I'll be keeping a close eye on it.

And, the transit center itself seems like it will be pretty nice, with a beautiful (I hope) blocks-long public park atop the terminal, that should be very fine on sunny afternoons.

But the city is definitely a warren of canyons. You'd never believe you were just a few blocks from some of the most beautiful scenery in the world (the Golden Gate, the Bay, Alcatraz and Angel islands, etc.); you'd think you were somewhere around midtown-Manhattan, instead.

Which, I suppose, is sort of the point.

But here I am, and up they go.

Sunday, January 15, 2017

Stuff I'm reading, MLK Day edition

My, how time races by. I've been quite busy and not getting enough time to read.

Is there ever enough time to read?

  • CIDR 2017
    The biennial Conference on Innovative Data Systems Research (CIDR) is a systems-oriented conference, complementary in its mission to the mainstream database conferences like SIGMOD and VLDB, emphasizing the systems architecture perspective. CIDR gathers researchers and practitioners from both academia and industry to discuss the latest innovative and visionary ideas in the field.
  • Optimizing Space Amplification in RocksDB
    RocksDB is an embedded, high-performance, persistent keyvalue storage engine developed at Facebook. Much of our current focus in developing and configuring RocksDB is to give priority to resource efficiency instead of giving priority to the more standard performance metrics, such as response time latency and throughput, as long as the latter remain acceptable. In particular, we optimize space efficiency while ensuring read and write latencies meet service-level requirements for the intended workloads. This choice is motivated by the fact that storage space is most often the primary bottleneck when using Flash SSDs under typical production workloads at Facebook. RocksDB uses log-structured merge trees to obtain significant space efficiency and better write throughput while achieving acceptable read performance.
  • How and why the leap second affected Cloudflare DNS
    The root cause of the bug that affected our DNS service was the belief that time cannot go backwards. In our case, some code assumed that the difference between two times would always be, at worst, zero.
  • The Road to 2 Million Websocket Connections in Phoenix
    2 million is a figure we are pleased with. However, we did not quite max out the machine and we have not yet made any effort toward reducing the memory usage of each socket handler. In addition, there are more benchmarks we will be performing. This particular set of benchmarks was set exclusively around the number of simultaneous open sockets. A chat room with 2 million users is awesome, especially when the messages are broadcast so quickly. This is not a typical use case though.
  • Adaptive logging: optimizing logging and recovery costs in distributed in-memory databases
    This is a paper about the trade-offs between transaction throughput and database recovery time. Intuitively for example, you can do a little more work on each transaction (lowering throughput) in order to reduce the time it takes to recover in the event of failure. Recovery is based on information in logs, classically an ARIES-style write-ahead log, that records the values of data items.

    In the case of in-memory databases, you can also go the other way, and do a little less work when creating the logs (recording information for use in recovery) at the expense of longer recovery times, but gaining higher throughput. We can simplify recovery on the assumption that there is no need to undo the effects of uncommitted transactions – these existed solely in-memory and had not yet been persisted to disk.

  • Millions of Queries per Second: PostgreSQL and MySQL’s Peaceful Battle at Today’s Demanding Workloads
    The idea behind this research is to provide an honest comparison for the two popular RDBMSs. Sveta and Alexander wanted to test the most recent versions of both MySQL and PostgreSQL with the same tool, under the same challenging workloads and using the same configuration parameters (where possible). However, because both PostgreSQL and MySQL ecosystems evolved independently, with standard testing tools (pgbench and SysBench) used for each database, it wasn’t an easy journey.
  • Bitpacking and Compression of Sparse Datasets
    It turns out that gzipping after bitpacking yields a 1000x compression. Even on its highest compression settings, gzip was leaving a 8x compression on the table when applied to the raw data. It turns out that if you know the structure of your own data, you can very easily do much, much better than a generic compression algorithm. -- on both speed and compression.
  • The Real Reason Your City Has No Money
    All of the programs and incentives put in place by the federal and state governments to induce higher levels of growth by building more infrastructure has made the city of Lafayette functionally insolvent. Lafayette has collectively made more promises than it can keep and it's not even close. If they operated on accrual accounting -- where you account for your long term liabilities -- instead of a cash basis -- where you don't -- they would have been bankrupt decades ago. This is a pattern we see in every city we've examined. It is a byproduct of the American pattern of development we adopted everywhere after World War II.
  • Software Copyright Litigation After Oracle v. Google
    Oracle America has factored into at least four cases so far. One of these cases settled, one is on appeal, and the other two likely will be appealed in the near future. The latter two cases also involve patent claims, so appeals will be heard by the CAFC. (The CAFC has nearly exclusive appellate jurisdiction over cases with patent claims.) One can assume that the plaintiffs added the patent claims to ensure CAFC jurisdiction.
  • This is Fine: Engineering War Stories (and What We Learned) in 2016
    In the past an engineer would be tasked with a project, crawl into a dark hole, and come out days, weeks or months later clutching their precious code. Sometimes this worked out really well but other times it was disastrous.
  • The Inside Story of BitTorrent’s Bizarre Collapse
    sometimes technologies are not products. And they’re not companies. They’re just damn good technologies.
  • “Side Hustle” as a Sign of the Apocalypse
    And WTF has happened to our culture when we just take it as fact that everyone needs to have multiple jobs and work as a cab driver and rent out every square inch of space in their apartment and be a task rabbit gopher who waits in line for tickets when they’re not walking dogs or temping and we all just chalk it up to “progress”??? In the old days, this meant your life was falling apart. Now it just means you’re part of “the sharing economy.”
  • The Chemistry Behind Your Home’s Water Supply
    We take for granted the water that comes out of the taps in our home when we turn them on – but a lot of work goes into getting it there. Chemistry, too, has a hand in making sure that the water is safe to drink. Here, we take a look at the water treatment process, and in particular the chemicals used to get clean drinking water to your tap.

Saturday, January 14, 2017

And the water from the mountains flowed into the bay

Despite days of rain, ‘high-tide groupies’ seek out king tides

Many among the curious onlookers snapped pictures, splashed in the water, or just stared at the curious phenomenon around 10:15 a.m., when the water crested in front of Pier 14.

It was day-two of a three day stretch of the unusually high water that swelled around the Bay Area and spilled onto low-lying coastal areas, prompting the National Weather Service to issue a coastal flood advisory.

Thursday, January 12, 2017

And the atmospheric river flowed into the reservoirs

20 inches of rain, 12 feet of snow finally end 5-year drought in N. California

Stations up and down the Sierra mountain chain reported twice the amount of normal rain and snow for this time of year after snowstorms doubled the vital snowpack there that provides the state with much of its year-round water supply.

U.S. Drought Monitor: California

A plethora of Pacific storms and moisture slammed into California and most of the West, dumping copious amounts of precipitation on the northern two-thirds of the state and Sierra Nevada. This very wet week maintained the great start to the Water Year (since Oct. 1) across the West where NRCS SNOTEL basin average precipitation was above or much above normal at nearly every major basin while basin average snow water content was at or above normal in most Western basins. With more than a foot of precipitation falling on the Sierra Nevada (locally 20.7 inches at Strawberry Valley, CA), most major reservoirs were at or above its Jan. 10 historical average, USGS monitored streams were at near or record high flows, Jan. 10 state snow water content was at 135%, and the Northern Sierra 8-station, San Joaquin 5-station, and Tulare Basin 6-station precipitation indices topped their wettest previous year as of Jan. 10.

A plethora of storms. Imagine that!

Even though my ferry was canceled one day, and there's definitely water everywhere, it's a merry, merry sight.