michaelochurch explains, “Industrial management has a 200-year history of successfully adding value by reducing variance, because in a concave world, low variance and high output go together. In a convex world (as in software) it’s the opposite: the least variance is in the territory where you don’t want to be anyway. #Convexity is a massive game-changer. It renders the old, control-based, #management regime worse than useless; in fact, it becomes counterproductive.” #software-engineering
#examples of doing #image-processing #registration with #Python and #RANSAC on starfield images, which allow the use of a particularly simple feature detector ☺ #dsp
a clever way to get Erlang-style IO lists in #Haskell by using ShowS, which uses partial application to build up the eventual concatenation operation, thus avoiding expensive repeated appends to the end of a list. I guess in effect you represent a string as a function of an output parameter, so concat is λx.λy.λprint.(print x; print y)? ☺ With the usual approach to representing closures, this is not unreasonably inefficient: a normal IO list using conses might need two pointers per cons (or three if you’re not using BiBOP or pointer tagging) while a concatenation closure needs only three, and also gives you extensibility to arbitrary text-generation functions.
Online #algorithms in high-frequency #trading (2013). This might be helpful in figuring out what kinds of things HFT algorithms can’t do in small or bounded memory, and therefore what exploitable market inefficiencies might still exist at the subsecond timescale.
Oh hey, Jane Street released #inotify bindings for #OCaml! Their documentation kind of sucks still.
some notes on the new features of the #OCaml 4.00.0 release in 2012, including much better #GDB integration and some new #performance optimizations, including in particular infix function application operators like Haskell's $ and flip $.
Oh my, the #OCaml Batteries Included package includes #algorithms and data structures like finger trees, dynamic arrays, abstract iteration (“enumeration”), UTF-8, and serialization (“marshal”). No inotify though.
A #paper about #trading: “An Optimization-Based Framework for Automated Market-Making (2011)”, reducing market-making to convex #optimization, and by relaxing the convex hull you can get the #algorithms to be computationally tractable without sucking too badly.
Another #trading #algorithms #paper by the same authors as the 2011 paper: “Efficient market making via convex optimization, and a connection to online learning (2012)”. Talks about Arrow-Debreu #prediction-markets as a market-based probability estimator and draws a connection to online #machine-learning algorithms, and briefly links to computational #complexity results. The reason the authors are interested in automated market-making seems to be that it is needed to make #prediction markets over complex outcome spaces feasibly liquid.
The 2012 #trading #algorithms #paper about convex optimization.
a bit about high-frequency #trading as a #DSP problem. Also explains adverse selection. I like the quote, “When a market maker trades with an informed investor, he is taking the wrong side of that trade.” I’m skeptical of his explanation about thresholds; wouldn’t you avoid that with hysteresis?
On the adversarial multi-armed #bandit problem.
#matplotlib has a pyplot.xkcd() function to reduce the perceived seriousness of #visualization results! This is awesome! I think it just uses Humor Sans or Comic Sans for the text though. And it’s not in the old version of matplotlib I’m using.
This #tutorial #introduction to A* #pathfinding #search for #video-games programming, by @redblobgames, has interactive demonstrations! I don’t remember those from before! The #UX somewhat lacks affordances to indicate that the things in the illustrations are draggable, so the text has to tell you, but in many other ways, these are among the best #visualization of #algorithms I’ve ever seen.
A bunch of #paper abstracts about, among other things, #search algorithms and multi-armed #bandit algorithms in the context of #management, i.e. operations research.
Code for several standard #algorithms for the multi-armed #bandit problem, in #Python, #Julia, and #Ruby.
A psychology textbook #ebook chapter about #nonparametric #statistics which omits even mentioning Kolmogorov-Smirnov Apparently, that’s because this is purely about nonparametric statistics on ordinal data, and the K-S test is really only applicable to data on at least an interval scale of measurement.
The idea of levels of scale of measurement: nominal, ordinal (ordering is possible), interval (distances are meaningful), and ratio (there is a nonarbitrary zero, so ratios are meaningful).
Tips for #shell scripting, including the ever-helpful “things I wish I’d known”.
Another #spreadsheet with functions for #cp4e.
Different implementations of #djb’s daemontools.
The #inflation calculator from the BLS using the #CPI, which is now harder to access than before.
3773 of the 19383 two-input ternary #logic gates are universal; the proportion of universal N-ary two-input gates goes to 1/e as N→∞.
Using #OW2-ASM to invoke #invokevirtual from #Java.
This looks an awful lot like dstaff’s house!
How to use #tags in #org-mode. They use colons instead of hash marks like everybody else.
Debugging the Sun #JVM with #GDB from a core dump crash file.
The #hsdis #HotSpot #JVM disassembler for #Java #performance work.
How to control nondeterminism and make #performance #profiling possible on the #Java #JVM. This has been a real problem for me in developing #DyCSV.
Troubleshooting guide for #Java SE 6. Explains what all those -XX flags mean and jhat, jmap, jps, jstack, hprof, etc.
Books to read to get an #education.
“High-level languages are slow” because “more allocations means more time spent collecting garbage.” Sylvan is one of the thinkers who is leading me to articulate the difference between the different models of memory provided by different languages: object graphs vs. object embedding vs. #parallel-arrays vs. logic programming and linear types and whatever else. #performance
Commentary on Sylvan’s #performance gripes about object-graph languages.
Luca Cardelli, who probably knows more about #OO than anybody else, explains what the problems were with it in this #paper from around 1997.
difficulties with #OO in e.g. #C++ when programming #video-games for the #PlayStation 3 in 2009. Unfortunately these are slides. Basically the author’s claim is that C++ encourages memory layouts that have terrible #performance on modern hardware. Very concrete, with lots of C++ #examples and PowerPC #asm disassembled from them of 3-D programming and diagrams of #cache line evictions and whatnot. Basically his recommended cure is to use #parallel-arrays for most things and “flat” contiguous level-order tree linearizations for tree data.
Project #Lombok supports automatic bytecode generation of Builders for your #Java classes with a higher-order annotation, so you can say Person.builder().name("Adam Savage").city("San Francisco").job("Mythbusters").job("Unchained Reaction").build(); without having to write a Builder class or its name, city, job, or build methods.
A 2010 introduction to Project #Lombok.
The Böhm and Jacopini theorem about goto.
Michał Zalewski (#lcamtuf) very quickly found crash bugs in #SQLite with #afl.
A #FreeBSD person wrote this April blog post about “Inclusionism in the *BSD community or, Why digital colorblindness can’t work” and deleted it by August, presumably because the #social-justice aspects resulted in a huge flamewar. In fact, they deleted their entire blog.
“a performative understanding of knowledge requires us to understand and account for the fact that epistemic practices are inherently ethical practices” #feminism and free software
review of research on #typography to see whether ragged-right or justified alignment (text-align: justify) is more readable.
#Probabilistic-programming allows you to condition the execution of a nondeterministic program on observations, reasoning backwards from the observations to the conditional distribution of the original stochastic variables, thus allowing you to express #statistics models in a Turing-complete form. (However, some PPS languages are not Turing-complete.) Well-known languages are Venture, Figaro, and Berkeley’s BLOG, but there are also Church, Haraku, Chimple/Dimple, Stan, and Infer.net. #toread
an #introduction to “#Stochastic-calculus, the branch of mathematics that is most identified with financial engineering and mathematical finance” explaining Brownian motion, martingales, stochastic integrals, etc. #trading #paper #toread
the Michigan Micro Mote is based on the 2008 Phoenix processor #hardware, which is a bit under a mm² and uses 500pW on average because its standby current is 2nA. You can get 20nW from a 1mm² solar cell. #smartdust
#marijuana seizures at the US-#Mexico border dropped by a third from 2011 to 2014. #legalization #politics
#silentbicycle’s Tangram project is a #blob store that stores data with his #content-addressable #storage Jumprope library, which uses #hashsplitting to chunk the data, much like #bup. Written in #Lua on top of #SQLite 3.
#Agile development practices focus on delivering functionality, but a #lean-startup needs to focus more on validating customer demand, so #startups shouldn’t use agile development? Because you need to focus on learning velocity instead of delivery velocity? Dude, that means you’re DOING AGILE DEVELOPMENT WRONG, because agile development is ALL ABOUT learning.
The sexual harassment and “sexual predator” allegations against librarian #Joe-Murphy by nina de jesus and Lisa Rabey are here retracted and admitted to be false, about ten months after they were originally made.
an article about librarian #Joe-Murphy and the apparently completely unfounded, and later retracted, sexual harassment and “sexual predator” allegations against him, which he lost his job over.
Wikipedia used to have a “featured sounds” curation process that cataloged really awesome downloadable #cc #audio, including a lot of #music, but this process ended in 2011.
SuperCSV is a stream-based #Java #CSV library with POJO support and data conversion. It still uses dynamic dispatch to figure out what to do after each field, though, making it slow.
The Apache Commons #CSV #Java library kind of sucks at performance but its API is not terrible.
“P-values are not error probabilities”, with an overview of the #history of #statistics with Fisher’s evidential p value contrasted with the Neyman–Pearson α.
#video demonstrating and explaining the construction of what surely must be the world’s most frugal large homemade #CNC #machining #hardware. Roughly 100μm repeatability over 50×50 cm. He uses threaded bars (Allthread) lubricated with used motor oil for the linear actuator, bare cartridge bearings as wheels. He uses low-power and therefore very slow (2mm/s) DC motors with an optical quadrature encoder (perhaps salvaged from a printer) for feedback, controlled by H-bridges from Arduino. Plastic tube couplings for U-joints on the threaded rods. A goofy little C program on his Linux box to control the Arduino. Calibrated against an inkjet printer. Engraves glass with a diamond milling cutter mounted in a flexible coupling to void breaking things. Cuts a Styrofoam wing cross-section with a soldering iron. Cuts gears from acrylic or aluminum with a milling cutter (like a Dremel mounted with hose clamps). Superb how-to video. By Norbert Heinz.
Norbert Heinz’s home page, a physicist. You can drive his robots remotely!
Norbert Heinz’s page about his very frugal large homemade #CNC machine.
another cheap #CNC machine made with drawer slides for the linear translation, and mostly lots of plywood with some square tube stock to stiffen it. Big NEMA steppers though. Hose-clamp U-joints made with fuel line. Noisy as hell. He’s got a Black-and-Decker clone of a Dremel tied down with hose clamps in the middle. More Allthread for the linear actuators. Powering the motors off cat-5 cable. #video
#Video of Dan #Gelbart's uniflow #steam-engine #engineering #hardware
Dan #Gelbart’s description of his 2010 #steam-engine. (Or maybe it's Tom Kimmel’s.) Includes a sort of zero-#TCE construction to keep the inside diameter of the cylinder constant: when the 440C stainless-steel cylinder expands, the graphite-filled #polyimide (Vespel) liner tries to expand even more, but can only do so toward the inside of the cylinder, thus counteracting the cylinder’s expansion. The piston itself is made of electroless-nickel-plated #Invar, which really seems like overkill, but whatever. The tabletop model reaches 10% efficiency.
A #refractory #ceramic (1850° or so), typically sintered with spark plasma sintering. Gelbart’s #steam-engine uses it for the intake valve because it's more resistant to being pitted by the steam than steel is. #materials
#Polar-3D is a #CNC #3D-printer using polar coordinates to get multiple print heads, which don’t have to move, and avoid linear actuators and linear bearings entirely, except for a Z-axis which raises and lowers the whole table, including the motors. Also, the platters provide precisely known movement for a laser #3D scanner, thus avoiding #photometric-stereo and the like. #video #manufacturing
The home page of the #Polar-3D #3D-printer.
#news cellphone #video of a large #explosion in #Tianjin yesterday
#News: the large #explosion in #Tianjin yesterday. A series of chemical tanks in the container terminal caught fire and exploded, killing all of the thousands of people within a one-kilometer radius. 2.9 on Richter.
Another #news #video montage of the #Tianjin #explosion. You can see blast shrapnel from a wall on its way to obliterate the first cameraperson.
“Server relief”: the client computes the #scrypt #hash of your #security #password, then sends the server a salted SHA-256 of it.