Archive for June 2024

Saturday 22 June 2024

New top story on Hacker News: Ask HN: Advice for Leading a Software Migration?

New top story on Hacker News: Ask HN: Advice for Leading a Software Migration?

Ask HN: Advice for Leading a Software Migration?
14 by drekipus | 10 comments on Hacker News.
Hey HN, I'm about to take lead of a decent sized software migration at work. (From V1 of some subsystem, to v2, both in house. We want to deprecated and eventually remove V1 totally) For 8 of our clients, totalling about 16 million customers. I don't have too many details to share, as I don't know what's relevant. But I'm asking if anyone has any advice or recommended reading regarding such? One book that is really inspiring me about it is "how big things get done" by Bent Flyvbjerg and Dan Gardner. In it, there's some key bits of advice such as * Think slow, ask fast, and mitigate long tailed risks. * Compartmentalize and stick to repeated processes. "Build with LEGOs" * Look around at other projects of similar nature. The last point is why I'm here, as I know some of you have been in the game for longer than I have, so feel free to share experiences that you might think is relevant, if you'd like.

June 22, 2024 0

Friday 21 June 2024

Thursday 20 June 2024

Wednesday 19 June 2024

Tuesday 18 June 2024

New top story on Hacker News: Show HN: Billard – Generate music from ball collisions in 2D space

New top story on Hacker News: Show HN: Billard – Generate music from ball collisions in 2D space

Show HN: Billard – Generate music from ball collisions in 2D space
34 by bambax | 5 comments on Hacker News.
Hello HN! Here's Billard. It combines music and physics into a unique creative tool, as I explore various unconventional methods for generating music. Most traditional music composition tools revolve around the idea of a repeatable pattern. Billard is a webapp that never repeats itself. It generates music automatically based on the collisions of balls in a 2D space. Collisions trigger notes (or chords) in a given key. One can add balls or move them (y-position is pitch); the app remembers its state between reloads; or it can be reset with the 'init' button on the top left. Gravity can be adjusted in real time to change the behavior of the balls. It owes a lot of inspiration to Brian Eno and Erik Satie (inventor of musique d'ameublement , or "furniture music"). Some may think the lack of pattern makes it not musical enough -- but this lets it be listened to —and watched— for a while without boredom. The webapp is made using plain JavaScript. (All SVG icons were made 'by hand'.) It uses Tone.js only for triggering piano samples. Beyond piano, it's MIDI-enabled and works well at slow speed with haunting, dark synth sounds. Hope you like it!

June 18, 2024 0

Monday 17 June 2024

Sunday 16 June 2024

Saturday 15 June 2024

Friday 14 June 2024

Thursday 13 June 2024

Wednesday 12 June 2024

Tuesday 11 June 2024

Monday 10 June 2024

Sunday 9 June 2024

New top story on Hacker News: Is KDB a sane choice for a datalake in 2024?

New top story on Hacker News: Is KDB a sane choice for a datalake in 2024?

Is KDB a sane choice for a datalake in 2024?
16 by sonthonax | 16 comments on Hacker News.
Pardon the vague question, but KDB is very much institutional knowledge hidden from the outside world. People have built their livelihoods around it and use it as a hammer for all sorts of nails. It's also extremely expensive and written in a language with origins so obtuse that it's progenitor APL needed a custom keyboard laden with mathematical symbols. Within my firm, it's very hard to get an outside perspective, the KDB developers are true believers in KDB, but they they obviously don't want to be professionally replaced. So I'm asking the more forward leaning HN. One nail in my job, is KDB as a data-lake and I'm being driven nuts by it. I write code in Rust that prices options. There's a lot of complex code involved in this, I use a mix of numeric simulations to calculate greeks and somewhat lengthy analytical formulas. The data that I save to KDB is quite raw, I save the market data and derived volatility surfaces, which are themselves complex-ish models needing some carefully unit-tested code to convert in to implied vols. Right now my desk has no proper tooling for backtesting that uses our own data. And I'm constantly being asked to do something about it, and I don't know what to do! I'm 99% sure KDB is the wrong tool for the job, because of three things: - It's not horizontally scalable. A divide and conquer algo on N<{small_number} cores is pointless. - I'm scared to do queries that return a lot of data. It's non trivial to get a day's worth of data. The query will just often freeze, it doesn't even buffer. Even if I'm just trying to fetch what should be a logical partition, the wire format is really inefficient and uncompressed. I feel like I need to engineering work for trivial things. - The main thing is that I need to do complex math to convert my raw data, order-books and vol-surfaces into useful data to backtest. I have no idea how do do any of this in KDB. My firm is primarily a spot desk, and while I respect my colleagues, their answer is: > Other firms are really invested in KDB and use KDB for this, just figure it out. I'm going nuts because I'm under the assumption that these other firms are way larger and have teams of KDB-quants doing the actual research. While we have some quant traders who know a bit of KDB but they work in the spot side with far more simple math. I keep on advocating for some Parquet style data-store with Spark/Dask/Arrow/Polars running on top of it that can be horizontally scaled and most importantly, with Polars, I can write my backtests in Rust and leverage the libraries I've already written. I get shot down with "we use KDB here". I just don't know how I can deliver a maintainable solution to my traders with this current infrastructure. Bizarrely, and this is a financial firm, no one in a team of ~100 devs has ever touched Spark style tech other than me here. What should I do? Are my concerns overblown? Am I misunderstanding the power of KDB?

June 09, 2024 0

Saturday 8 June 2024

Friday 7 June 2024

Thursday 6 June 2024

Wednesday 5 June 2024

Tuesday 4 June 2024

Monday 3 June 2024

New top story on Hacker News: Show HN: I made a tiny camera with super long battery life

New top story on Hacker News: Show HN: I made a tiny camera with super long battery life

Show HN: I made a tiny camera with super long battery life
54 by davekeck | 25 comments on Hacker News.
Hey HN! A few years ago someone kept trying to steal my motorcycle, so I decided to make a small camera with really long battery life to catch them. The hardware/software is totally open source, but the companion app only supports macOS currently. (I'm a big fan of native apps, and didn't want to block releasing on Linux/Windows support.) I wrote some blog posts about the process: PCB design: https://ift.tt/NRbLDzy Enclosure design: https://ift.tt/YbET4Vo Image pipeline: https://ift.tt/UdJfTom Rainproofing: https://ift.tt/vlVJZoI Source: https://ift.tt/kCmrQ4K

June 03, 2024 0

Sunday 2 June 2024

Saturday 1 June 2024