Using Google BigQuery with Programmatic SQL

An often encountered Antipattern in Python code are complex queries composed with the help of string substitutions and concatenations. SQLAlchemy Core allows you to generate queries in a programmatic way for many SQL databases like BigQuery.

more ...


Finally! Bayesian Hierarchical Modelling at Scale

For a long time, Bayesian Hierarchical Modelling has been a very powerful tool that sadly could not be applied often due to its high computations costs. With NumPyro and the latest advances in high-performance computations in Python, Bayesian Hierarchical Modelling is now ready for prime time.

more ...

Honey, I shrunk the target variable

Feature engineering takes up a huge part in the work-life of a data scientist. Sometimes this doesn’t stop at features but also the target variable itself is transformed leading to all kinds of unexpected consequences. In this post, you will learn about common pitfalls, how a transformation can affect the error measure, the math behind it, and even how all this can be used to your advantage.

more ...

More Efficient UD(A)Fs with PySpark

With the release of Spark 2.3 implementing user defined functions with PySpark became a lot easier and faster. Unfortunately, there are still some rough edges when it comes to complex data types that need to be worked around.

more ...