Luciano Scarpulla

How I gained my first 100+ stars on GitHub

2024-06-26T00:00:00+00:00

I wanted to reflect on how I got my first 100+ stars on a GitHub repository.</p>

Awesome lists</h2>
What are awesome lists?</p>

An awesome list is a list of awesome things curated by the community. There are awesome lists about everything from CLI applications to fantasy books.</p> </blockquote>
I really like browsing GitHub for 'awesome lists', one of my favourites is awesome-selfhosted</a> which contains a list of selfhosted tools and resources.</p>
I wanted to create my own collection of free and open-source aviation projects that I know of. While working in aviation, I always had those sources in mind but I never kept a written collection of it, so I am doing this also for my personal benefit.</p>
As of the writing of this post, my list contains almost 20 entries (mostly data sources). In the coming days I intend to expand it further with more data and tools.</p>
I got spotted</h2>
Shortly after publishing the list, I got spotted by Cyber Detective</a>, an account which specializes on Open Source Intelligence (OSINT). They shared a link of my repository on X which then got 8k views.</p>
I was quite surprised by the rapidity at which my list (just a simple README file) got discovered. It is probably due to the 'osint' and 'awesome-list' tags that I put on GitHub.</p>
Growth so far</h2>
</p>
My list was featured in other people's awesome-list which is nice to see. To make it more "attractive", I decided to add a Python script that will look for any data source present in that file and append a table containing information for each data source.</p>
I even added a GitHub action to run the script every day, but I recently turned that off as I was not happy with the way it commits the repository.</p>
Overall, I think the key to success was the simplicity of the content as well as a good series of tags to make it more 'discoverable' on GitHub.</p>

My first experience with Coursera

2024-04-06T00:00:00+00:00

Why Coursera?</h2>
Apart from University, most of the professional learnings I had were via other "non-conventional" means such as Youtube videos and plain-text tutorials.</p>
Learning this way, can have many advantages, for example:</p>

You can learn at your own pace</li>
You can watch and repeat over and over without being pressed by deadlines and exams</li>
You manage your own curriculum by choosing the topics you actually want to learn</li> </ul>
However, it can become hard to prove the knowledge and skills you acquired online without any kind of "valid proof" or certificate. Sometimes, the best way of proving your skills is to just do it, and that is why I think showcasing what you can do is very important.</p>
Recently, I was encouraged to take on a course for then getting a certificate. I went on Google and searched a topic I have known and learned but which I would like to further develop and prove my knowledge with a certification.</p>
Pytorch is a library used for Machine Learning which is a hot topic that hasn't stopped trending. Coursera offered an intermediate course by IBM on it so I decided to give it a try.</p>
First impressions</h2>
Compared to what you can usually find on Youtube and other websites, the course seems to be very "cheap" with a synthetic voice over and poorly made slides. But the explanations and concepts are still very simple and easy to understand. I was just surprised that the average quality of the course is not in par with what you would normally have on a free platform.</p>
Is it actually worth it?</h2>

I guess no.</p> </blockquote>
But I did a 7 days free trial and managed to complete the course within that timeframe.</p>
I did learn new things and I think that the overall experience is decent. But I don't think it is worth the starting price of 45 Euros per month.</p>
What next</h2>
I still cannot find a good course on AI which is better than what you can find freely online. I guess the price you pay is just for a fancy certificate to put on LinkedIn?</p>

I was wrong about Rye

2024-03-14T00:00:00+00:00

While talking about Rye, in the previous post I wrote that the lack of implementations for VScode or Pycharm had made me reluctant to adopt it full time.</p>

By "lack of implementations" I meant that there was no python environment finder suited for Rye's virtual environment. I probably did not use Rye properly in the first time, since Rye's virtual env is literally</em> a Virtualenv, and so it can be discovered by all majors IDEs.</p>

Somehow, I first thought Rye had its own virtual environment type!</p>

Since finding this out, I now intend to use it full-time and will post more about it whenever needed.</p>

My tips for using it is to make sure to rye sync</code> after every rye add</code> since this last command does not install the actual packages that will be listed on the project configuration itself. Although I like this behaviour, I would love to have an option to default rye sync</code> after every rye add</code> (similar to cargo add</code>).</p>

Also, uv</code> integration does not come as a default. It needs to be enabled in the global rye configurations!</p>


The airport compass
2024-02-25T00:00:00+00:00
A few years ago, I got inspired by a very interesting figure published on a paper by Geoff Boeing</a> called "Urban spatial order: street network orientation, configuration, and entropy"</a>.</p>
This figure represents the distribution of the orientation of every street of major cities. I like how it gives a quick representation of each city's layout and how evident are the various levels of entropy.</p>
Shortly after seeing this chart, I immediately thought of replicating this view for airports' traffic worldwide.</p>
Sources and code</h2>
For this project I used two major open-source datasets:</p>

The airlines routes data comes from @jpatokal</a>'s OpenFlights repository</li>
The airports data is from @davidmegginson</a>'s OurAirports repository</li>
</ul>
I made the source-code available on my GitHub here</a>.</p>
Outcomes</h2>
I first generated the top 15 airports by number of destinations. Each chart (or compass) displays the distribution of outbound flights and aligns those for each initial bearing. The longer the yellow bar, the more frequent is that bearing. For example, if an airport has most of the departures pointing West (or 270°), you would see the longest bar pointing left.</p>
</p>
Some airports like New York JFK and London Heathrow (LHR) represent an important hub for transatlantic flights. In these charts, we can see how LHR has a long bar towards West (towards the North America) and JFK has one towards East (Europe).</p>
</p>
Other airports' compasses, instead, are majorly driven by the geographical constrains of their location. Tokyo's Narita Intl. Airport is a very good example to illustrate this case. It has virtually no routes pointing eastwards! This is because Japan has the Pacific Ocean to its east side and most of the routes connecting Japan to North America are flying with an initial bearing pointing North! (Since this would be the shortest great circle path)</p>
</p>
How-to</h2>
As I said earlier in this post, I am using two freely available datasets and published the source code to replicate these charts. Everything is made entirely using Python:</p>

To merge and transform the data I used pandas</a></li>
matplotlib</a> is used for the visualization</li>
</ul>
Data</h3>
First, I set a constant to represent the number of airports I want to have in the final view. In this example, we are going to start with 30. The csv data can be conveniently loaded directly from GitHub using the read_csv</code> method.</p>
import</span> pandas</span> as</span> pd</span></span>
</span>
TOP_AIRPORTS</span> =</span> 30</span></span>
</span>
# Getting airlines' routes data:</span></span>
routes</span> =</span> pd.read_csv(</span></span>
    "https://raw.githubusercontent.com/jpatokal/openflights/master/data/routes.dat"</span>,</span></span>
    usecols</span>=</span>[</span>0</span>,</span> 2</span>,</span> 4</span>],</span></span>
    names</span>=</span>[</span>"airline"</span>,</span> "origin"</span>,</span> "destination"</span>],</span></span>
)</span></span></code></pre>
This first dataset doesn't come with headers, hence why we need to use the names</code> argument to specify the column names. Also, we are just interested in knowing the origin and destination airport of each route (Column 2 and 4), I also added the airline IATA code (Column 0) because I was curious, but not needed for this project.</p>
We are also not interested in displaying all airports on this dataset. To do so, we can get the Top airports by grouping the data, ordering it and accessing only until the TOP_AIRPORTS</code> constant number:</p>
# Selecting only top airports</span></span>
top_airports</span> =</span> (</span></span>
    routes.groupby(</span>"origin"</span>)</span></span>
    .destination.count()</span></span>
    .sort_values(</span>ascending</span>=</span>False</span>)</span></span>
    .index[:</span>TOP_AIRPORTS</span>]</span></span>
)</span></span>
</span>
routes</span> =</span> routes.loc[routes.origin.isin(top_airports)]</span></span></code></pre>
Now, we are just having data about each airlines' routes. But in order to plot the data into small compasses, we will need to calculate the initial bearing of each airport pair. And in order to do so, we also need the coordinates of each airport:</p>
# Getting airports data:</span></span>
airports</span> =</span> pd.read_csv(</span></span>
    "https://raw.githubusercontent.com/davidmegginson/ourairports-data/main/airports.csv"</span>,</span></span>
    usecols</span>=</span>[</span>"ident"</span>,</span> "iata_code"</span>,</span> "latitude_deg"</span>,</span> "longitude_deg"</span>],</span></span>
)</span></span></code></pre>
While looping for origin</code> and destination</code> we also make sure that the columns in the airport dataset are suffixed with "_origin" and "_destination" respectively. We can now calculate the bearing (angle) for each pair:</p>
import</span> numpy</span> as</span> np</span></span>
</span>
# Getting initial bearing for each route:</span></span>
df[</span>"angle"</span>]</span> =</span> np.degrees(</span></span>
    np.arctan2(</span></span>
        df.latitude_deg_destination</span> -</span> df.latitude_deg_origin,</span></span>
        df.longitude_deg_destination</span> -</span> df.longitude_deg_origin,</span></span>
    )</span></span>
)</span></span></code></pre>
Numpy can do all the math for us with a resulting column "angle" which contains all angles for each pair.</p>
Since we would want to have a sense of the "distribution" of routes for each angle, we can make a resulting dataframe by doing a final aggregation:</p>
# Aggregating final data:</span></span>
res</span> =</span> df.groupby([</span>"origin"</span>,</span> "angle"</span>]).destination.count()</span></span></code></pre>Visualization</h3>
We can dynamically create the layout of our "poster" by dividing the total number of airports by the number of columns we want to have:</p>
ncols</span> =</span> 5</span></span>
nrows</span> =</span> len</span>(top_airports)</span> //</span> ncols</span></span></code></pre>
Now we can set up the figure and axes of our plot:</p>
fig, axes</span> =</span> plt.subplots(</span></span>
    nrows</span>=</span>nrows,</span></span>
    ncols</span>=</span>ncols,</span></span>
    subplot_kw</span>=</span>dict</span>(</span>projection</span>=</span>"polar"</span>),</span></span>
    figsize</span>=</span>(ncols</span> *</span> 1.9</span>, nrows</span> *</span> 2.6</span>),</span></span>
)</span></span></code></pre>
We can fill each axis with a for-loop:</p>
for</span> ax, airport</span> in</span> zip</span>(axes.flatten(), top_airports):</span></span>
    airport_data</span> =</span> res.loc[airport]</span></span>
    ax.bar(</span></span>
        np.radians(airport_data.index),</span></span>
        airport_data.values,</span></span>
        width</span>=</span>0.3</span>,</span></span>
        color</span>=</span>"C2"</span>,</span></span>
        alpha</span>=</span>0.5</span>,</span></span>
    )</span></span>
</span>
    ax.set_title(</span>f</span>"</span>{</span>airport</span>}\n</span>"</span>,</span> fontdict</span>=</span>dict</span>(</span>fontweight</span>=</span>"bold"</span>))</span></span>
    ax.set_yticks([])</span></span>
    ax.set_rlabel_position(</span>90</span>)</span></span>
    ax.set_xticklabels([</span>"E"</span>,</span> ""</span>,</span> "N"</span>,</span> ""</span>,</span> "W"</span>,</span> ""</span>,</span> "S"</span>,</span> ""</span>],</span> fontdict</span>=</span>dict</span>(</span>fontsize</span>=</span>10</span>))</span></span></code></pre>
Here, axes.flatten()</code> conveniently "flattens" out the nested array that axes</code> is. We don't need the ticks for the Y axis so we turn them off using ax.set_yticks([])</code>.</p>
The set_xtickslabels</code> currently raise a UserWarning</code> which we can safely ignore.</p>
fig.suptitle(</span>"Where is the traffic going?"</span>,</span> fontsize</span>=</span>25</span>,</span> fontweight</span>=</span>"bold"</span>)</span></span>
fig.text(</span>0.05</span>,</span> 0.005</span>,</span> "Made by Luciano Scarpulla"</span>)</span></span>
fig.tight_layout(</span>pad</span>=</span>3</span>)</span></span>
fig.savefig(</span>f</span>"Where is it going </span>{TOP_AIRPORTS}</span>.png"</span>,</span> dpi</span>=</span>300</span>)</span></span></code></pre>What's next?</h2>
I first developed this years ago and re-adapted my old code for this post. There are some changes and improvements that can be made:</p>

The data is grouped by each unique bearing. This can be fixed by simply rounding the angles, or distributing the angles into different buckets (e.g 130-140 degrees, 140-150, etc.)</li>
The image is a static outlook at airlines' routes of 2014. A seasonal outlook could be made so that you could notice the differences in the "orientation" in certain airports.</li>
</ul>
</p>


Switching to Rust as a Python developer
2024-02-02T00:00:00+00:00
Learning Rust for the past months has been one best learning experiences I have had since learning Python. If you have to pick a new language to learn in 2024, I would definitely recommend Rust!</p>
Here are the top features that I particularly enjoy as a Python developer!</p>
Why Rust?</h2>
You may want to know why I wanted to learn Rust out of the many other languages out there. First, I wanted to pick a lower-level language as I think it would nicely complete with my current knowledge of Python. By learning a lower-level language I can familiarize myself more with hardware resources and architectures as well as learning how to optimize for performance. As a Python developer, I kept looking more and more at rust-dependent code such as Polars</a> and Pydantic</a>, there is definitely a trend for applications to have a core backend written in Rust and using Python as a code interface. This is one of the main reasons I am choosing to learn Rust among many other great languages.</p>
Static typing</h2>
Although I really like the dynamic nature of Python, I have been starting to appreciate the comfort of having typehints in the code I write. Typehints improve code readability and blends with most code editors and lsps</strong>.</p>
Writing typehints in my Python code has actually saved me time! Every time I am using the wrong parameter for a function or when looking for a bug, I can always keep an eye for the warnings that wouldn't normally be there if I hadn't put typehints in the first place!</p>
All of this makes me code with more ease. Moving to a statistically typed language is exciting and brings me even more comfort because I know there is little room for silly errors. Which brings me to the next amazing feature Rust has.</p>
The compiler</h2>
I love how the compiler is the best code companion in Rust. After compilation, I am comfortable that there are not going to be any major issues since most of them were caught by the compiler. Rust makes sure that your script is safe to run. And if any errors are raise during compilation, these are presented in a very clear way and it even provide helpful solutions. As I learn my way through Rust, I often learn things just by compiling my scripts!</p>
Portability</h2>
Python is a great language which offers a lot of flexibility on what to develop and how. But sharing your Python app is sometimes difficult. You always need to consider the environment in which Python needs to be interpreted and run. Moving to a compiled language really made me discover the advantage of having a single binary file that can be executed in different machines with very few difficulties.</p>
Cargo</h2>
I tried various library mangers in Python. Although I mainly use anaconda</a> for my day-to-day development which suits most of my cases, I sometimes will use poetry</a> and I even start experimenting with rye</a>. But still, to this day, managing environments and packages in Python can be very difficult, especially because there could be many different ways to pack a library and each library will bring its own sometimes messy requirements. Some popular libraries are very easy to install in any Os but others can be very cumbersome! Cargo brings serenity to all of this. You can easily add libraries to your Rust project, compile it and run it without having to do any manual work. Learning Rust has been particularly pleasant for me especially because of Cargo</a>.</p>
Safety</h2>
More specifically Null safety and error handling. Yes - not necessarily memory safety since with Python I never really had to deal with memory leaks by leaving the garbage collector do its job. In Rust, there two main Enums</strong> that makes handling null values and errors very trivial. These are Option</code> and Result</code> respectively.</p>
In Python, you can convey that a function might return a value or None</code> with the Option</code> typehint or, more recently, with the notation | None</code>. But this still leave room for the usuals ValueError</code> or AttributeError</code> and leave the optional freedom to programer to handle null cases.</p>
Consider the following example in Python:</p>
import</span> random</span></span>
</span>
</span>
def</span> foo</span>() -></span> str</span> |</span> None</span>:</span></span>
    if</span> random.randint(</span>0</span>,</span> 10</span>)</span> ==</span> 1</span>:</span></span>
        return</span> "Hello"</span></span>
    else</span>:</span></span>
        return</span> None</span></span>
</span>
</span>
def</span> bar</span>(s:</span> str</span> |</span> None</span>) -></span> str</span>:</span></span>
    if</span> isinstance</span>(s,</span> str</span>):</span></span>
        return</span> s.upper()</span></span>
    return</span> ""</span></span>
</span>
</span>
random_val</span> =</span> foo()</span></span>
assert</span> bar(random_val)</span> in</span> (</span>"HELLO"</span>,</span> ""</span>)</span></span></code></pre>
Now, this is a very basic examples! There are chances for this program to run successfully but let's see why it may not.</p>
The function bar</code> is expecting a value of type str</code>, but since the function foo</code> may return a string or None</code>, line 8</code> will raise an AttributeError</code> since None</code> does not have the method .upper</code>. The rest of the code really won't be run until foo</code> returns "Hello"</code>.</p>
We can make this code safer</em> with one extra line of code:</p>
import</span> random</span></span>
</span>
def</span> foo</span>() -></span> str</span> |</span> None</span>:</span></span>
    if</span> random.randint(</span>0</span>,</span>10</span>)</span> ==</span> 1</span>:</span></span>
        return</span> "Hello"</span></span>
</span>
def</span> bar</span>(s:</span>str</span> |</span> None</span>) -></span> str</span>:</span></span>
    if</span> isinstance</span>(s,</span>str</span>):</span></span>
        return</span> s.upper()</span></span>
    return</span> ""</span></span>
</span>
random_val</span> =</span> foo()</span></span>
assert</span> bar(random_val)</span> in</span> (</span>"HELLO"</span>,</span>""</span>)</span></span></code></pre>
I love the approach that Rusts takes to handle this case using Option</code>. Here is how the previous example would look like written in Rust:</p>
Unfortunately</em>, we need to add a package to address the random behavior of the foo</code> function as Rust does not have it to the standard library.</p>
Here I am using Cargo to add the package rand</em>.</p>
cargo</span> add rand</span></span></code></pre>use</span> rand</span>::</span>{thread_rng,</span> Rng</span>};</span></span>
</span>
fn</span> foo</span>()</span> -></span> Option</span><</span>String</span>> {</span></span>
    let mut</span> rng</span> =</span> thread_rng</span>();</span></span>
</span>
    if</span> rng</span>.</span>gen_range</span>(</span>0</span>..</span>10</span>)</span> ==</span> 1</span> {</span></span>
        Some</span>(</span>String</span>::</span>from</span>(</span>"Hello"</span>))</span></span>
    }</span> else</span> {</span></span>
        None</span></span>
    }</span></span>
}</span></span>
</span>
fn</span> bar</span>(s</span>:</span> Option</span><</span>String</span>>)</span> -></span> String</span> {</span></span>
    s</span>.</span>unwrap_or</span>(</span>String</span>::</span>from</span>(</span>""</span>))</span>.</span>to_uppercase</span>()</span></span>
}</span></span>
</span>
fn</span> main</span>() {</span></span>
    let</span> random_val</span> =</span> foo</span>();</span></span>
    let</span> res</span> =</span> vec!</span>[</span>"HELLO"</span>.</span>to_string</span>(),</span> ""</span>.</span>to_string</span>()];</span></span>
    assert!</span>(res</span>.</span>contains</span>(</span>&</span>bar</span>(random_val)))</span></span>
}</span></span></code></pre>
With this code, foo</code> uses the Option</code> and it's variant Some</code> which literally means that there is a value, and None</code> - no value.</p>
This makes so that any function that uses the output of foo</code> would need to handle each variant of the Enum Option</code>.</p>
The same would be true for the Result</code> enum which has Ok</code> and Err</code> as a variant. With Rust you could simply use the unwrap_or</code> method to get the Ok</code> value or use a match pattern like this:</p>
use</span> rand</span>::</span>{thread_rng,</span> Rng</span>};</span></span>
</span>
fn</span> random_error</span>()</span> -></span> Result</span><</span>String</span>, ()> {</span></span>
    let mut</span> rng</span> =</span> thread_rng</span>();</span></span>
</span>
    if</span> rng</span>.</span>gen_range</span>(</span>0</span>..</span>10</span>)</span> ==</span> 1</span> {</span></span>
        Ok</span>(</span>String</span>::</span>from</span>(</span>"Result A"</span>))</span></span>
    }</span> else</span> {</span></span>
        Err</span>(())</span></span>
    }</span></span>
}</span></span>
</span>
fn</span> main</span>() {</span></span>
    let</span> val</span> =</span> random_error</span>();</span></span>
    match</span> val {</span></span>
        Ok</span>(result)</span> =></span> println!</span>(</span>"We safely got the result, {result}"</span>),</span></span>
        Err</span>(_)</span> =></span> println!</span>(</span>"We received an error!"</span>),</span></span>
    }</span></span>
}</span></span></code></pre>Final thoughts</h2>
I spent quite some words describing how good</em> Rust is and how bad</em> Python can get. But there is a whole new post that can be made on how good Python is compared to Rust, in terms of its rapid prototyping, learning curves, huge community, and much more! I won't stop using Python as I think it is a fundamental language for data analytics but I am ready to start experimenting with Rust for tools that would require more control and performance.</p>


Python tools I use
2024-01-10T00:00:00+00:00
As previously mentioned in this blog, Rust is becoming more and more used in Python libraries such as Pydantic and Polars. But recently we are also getting python tools written in Rust!</p>
Few months ago, I followed the development of Rye</a> as an alternative Python project and package manager. Later on, Ruff</a> made its debut as a linter and formatter substituting the popular flake-8</a> and black</a>.</p>
Thanks to their powerful backend written in Rust, these tools all provide a substantial upgrade in terms of performance and flexibility compared to the tools they are aiming to replace (up to 100x faster).</p>
Ruff</h2>
I found ruff format</code> to be the easiest tool to replace. I often bind the reformat command to a shortcut or to reformat while saving. But more often, I tend to reformat the whole project before committing. The performance of Ruff allows me to confidently reformat thousands of lines knowing that I will not need to wait long.</p>
Adopting ruff format</code> together with ruff check</code> in your pipelines can substantially reduces your actions time!</p>
Rye</h2>
Even before Astral's ruff and uv were available, I got Rye a try. Rye was the first attempt to implement a "Cargo for Python" - a 360 degrees project and package management at your fingertips. I immediately found Rye very easy to use and, to some extent, similar to Poetry! However, the lack of implementations for VScode (at the time of testing) or Pycharm had made me reluctant to adopt it full time.</p>
Uv</h2>
Uv is the latest addition to the Rust-born Python family tools. It is the fastest package manager out there and is able to quickly resolve most of your painful package dependencies. It is also a replacement for venv</code>, so you can quickly set up a new virtual environment for your Python project.</p>
After Uv was released, Astral also announced the takeover of Rye and implemented uv as the core package manager for Rye.</p>
What I use, what I am not using and what's next</h2>
I now fully use ruff on a daily basis both at work and for private projects. It is so easy to use that I often forget it is running.</p>
Due to the current lack of support for Pycharm (the IDE I use at work) I am not currently using either Rye nor Uv at work, but I am still using them on my local machine for personal projects.</p>
</p>
I look forward to see what Astral is developing next - perhaps a "Clippy for Python" or a rust-based static type checker to substitute mypy? Everything seems to be possible at this point.</p>

Luciano Scarpulla

How I gained my first 100+ stars on GitHub

My first experience with Coursera

What next</h2> I still cannot find a good course on AI which is better than what you can find freely online. I guess the price you pay is just for a fancy certificate to put on LinkedIn?</p>

I was wrong about Rye

The airport compass

How-to</h2> As I said earlier in this post, I am using two freely available datasets and published the source code to replicate these charts. Everything is made entirely using Python:</p>

Switching to Rust as a Python developer

Python tools I use

What next</h2>
I still cannot find a good course on AI which is better than what you can find freely online. I guess the price you pay is just for a fancy certificate to put on LinkedIn?</p>