LaForge's home page (Posts about openbsc)

Osmocom jenkins test suite execution

Harald Welte — Sat, 19 Aug 2017 16:00:00 GMT

Automatic Testing in Osmocom

So far, in many Osmocom projects we have unit tests next to the code. Those unit tests are executing test on a per-C-function basis, and typically use the respective function directly from a small test program, executed at make check time. The actual main program (like OsmoBSC or OsmoBTS) is not executed at that time.

We also have VTY testing, which specifically tests that the VTY has proper documentation for all nodes of all commands.

Then there's a big gap, and we have osmo-gsm-tester for testing a full cellular network end-to-end. It includes physical GSM modesm, coaxial distribution network, attenuators, splitter/combiners, real BTS hardware and logic to run the full network, from OsmoBTS to the core - both for OsmoNITB and OsmoMSC+OsmoHLR based networks.

However, I think a lot of testing falls somewhere in between, where you want to run the program-under-test (e.g. OsmoBSC), but you don't want to run the MS, BTS and MSC that normally surroudns it. You want to test it by emulating the BTS on the Abis sid and the MSC on the A side, and just test Abis and A interface transactions.

For this kind of testing, I have recently started to investigate available options and tools.

OsmoSTP (M3UA/SUA)

Several months ago, during the development of OsmoSTP, I disovered that the Network Programming Lab of Münster University of Applied Sciences led by Michael Tuexen had released implementations of the ETSI test suite for the M3UA and SUA members of the SIGTRAN protocol family.

The somewhat difficult part is that they are implemented in scheme, using the guile interpreter/compiler, as well as a C-language based execution wrapper, which then is again called by another guile wrapper script.

I've reimplemented the test executor in python and added JUnitXML output to it. This means it can feed the test results directly into Jenkins.

I've also cleaned up the Dockerfiles and related image generation for the osmo-stp-master, m3ua-test and sua-test images, as well as some scripts to actually execute them on one of the Builders. You can find related Dockerfiles as well as associtaed Makfiles in http://git.osmocom.org/docker-playground

The end result after integration with Osmocom jenkins can be seen in the following examples on jenkins.osmocom.org for M3UA and for SUA

Triggering the builds is currently periodic once per night, but we could of course also trigger them automatically at some later point.

OpenGGSN (GTP)

For OpenGGSN, during the development of IPv6 PDP context support, I wrote some test infrastructure and test cases in TTCN-3. Those test cases can be found at http://git.osmocom.org/osmo-ttcn3-hacks/tree/ggsn_tests

I've also packaged the GGSN and the test cases each into separate Docker containers called osmo-ggsn-latest and ggsn-test. Related Dockerfiles and Makefiles can again be found in http://git.osmocom.org/docker-playground - together with a Eclipse TITAN Docker base image using Debian Stretch called debian-stretch-titan

Using those TTCN-3 test cases with the TITAN JUnitXML logger plugin we can again integrate the results directly into Jenkins, whose results you can see at https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-ggsn-test/14/testReport/(root)/GGSN_Tests/

Further Work

I've built some infrastructure for Gb (NS/BSSGP), VirtualUm and other testing, but yet have to build Docker images and related jenkins integration for it. Stay tuned about that. Also, lots more actual tests cases are required. I'm very much looking forward to any contributions.

IPv6 User Plane support in Osmocom

Harald Welte — Tue, 08 Aug 2017 16:00:00 GMT

Preface

Cellular systems ever since GPRS are using a tunnel based architecture to provide IP connectivity to cellular terminals such as phones, modems, M2M/IoT devices and the like. The MS/UE establishes a PDP context between itself and the GGSN on the other end of the cellular network. The GGSN then is the first IP-level router, and the entire cellular network is abstracted away from the User-IP point of view.

This architecture didn't change with EGPRS, and not with UMTS, HSxPA and even survived conceptually in LTE/4G.

While the concept of a PDP context / tunnel exists to de-couple the transport layer from the structure and type of data inside the tunneled data, the primary user plane so far has been IPv4.

In Osmocom, we made sure that there are no impairments / assumptions about the contents of the tunnel, so OsmoPCU and OsmoSGSN do not care at all what bits and bytes are transmitted in the tunnel.

The only Osmocom component dealing with the type of tunnel and its payload structure is OpenGGSN. The GGSN must allocate the address/prefix assigned to each individual MS/UE, perform routing between the external IP network and the cellular network and hence is at the heart of this. Sadly, OpenGGSN was an abandoned project for many years until Osmocom adopted it, and it only implemented IPv4.

This is actually a big surprise to me. Many of the users of the Osmocom stack are from the IT security area. They use the Osmocom stack to test mobile phones for vulnerabilities, analyze mobile malware and the like. As any penetration tester should be interested in analyzing all of the attack surface exposed by a given device-under-test, I would have assumed that testing just on IPv4 would be insufficient and over the past 9 years, somebody should have come around and implemented the missing bits for IPv6 so they can test on IPv6, too.

In reality, it seems nobody appears to have shared line of thinking and invested a bit of time in growing the tools used. Or if they did, they didn't share the related code.

In June 2017, Gerrie Roos submitted a patch for OpenGGSN IPv6 support that raised hopes about soon being able to close that gap. However, at closer sight it turns out that the code was written against a more than 7 years old version of OpenGGSN, and it seems to primarily focus on IPv6 on the outer (transport) layer, rather than on the inner (user) layer.

OpenGGSN IPv6 PDP Context Support

So in July 2017, I started to work on IPv6 PDP support in OpenGGSN.

Initially I thought How hard can it be? It's not like IPv6 is new to me (I joined 6bone under 3ffe prefixes back in the 1990ies and worked on IPv6 support in ip6tables ages ago. And aside from allocating/matching longer addresses, what kind of complexity does one expect?

After my initial attempt of implementation, partially mislead by the patch that was contributed against that 2010-or-older version of OpenGGSN, I'm surprised how wrong I was.

In IPv4 PDP contexts, the process of establishing a PDP context is simple:

Request establishment of a PDP context, set the type to IETF IPv4
Receive an allocated IPv4 End User Address
Optionally use IPCP (part of PPP) to reques and receive DNS Server IP addresses

So I implemented the identical approach for IPv6. Maintain a pool of IPv6 addresses, allocate one, and use IPCP for DNS. And nothing worked.

IPv6 PDP contexts assign a /64 prefix, not a single address or a smaller prefix
The End User Address that's part of the Signalling plane of Layer 3 Session Management and GTP is not the actual address, but just serves to generate the interface identifier portion of a link-local IPv6 address
IPv6 stateless autoconfiguration is used with this link-local IPv6 address inside the User Plane, after the control plane signaling to establish the PDP context has completed. This means the GGSN needs to parse ICMPv6 router solicitations and generate ICMPV6 router advertisements.

To make things worse, the stateless autoconfiguration is modified in some subtle ways to make it different from the normal SLAAC used on Ethernet and other media:

the timers / lifetimes are different
only one prefix is permitted
only a prefix length of 64 is permitted

A few days later I implemented all of that, but it still didn't work. The problem was with DNS server adresses. In IPv4, the 3GPP protocols simply tunnel IPCP frames for this. This makes a lot of sense, as IPCP is designed for point-to-point interfaces, and this is exactly what a PDP context is.

In IPv6, the corresponding IP6CP protocol does not have the capability to provision DNS server addresses to a PPP client. WTF? The IETF seriously requires implementations to do DHCPv6 over PPP, after establishing a point-to-point connection, only to get DNS server information?!? Some people suggested an IETF draft to change this butthe draft has expired in 2011 and we're still stuck.

While 3GPP permits the use of DHCPv6 in some scenarios, support in phones/modems for it is not mandatory. Rather, the 3GPP has come up with their own mechanism on how to communicate DNS server IPv6 addresses during PDP context activation: The use of containers as part of the PCO Information Element used in L3-SM and GTP (see Section 10.5.6.3 of 3GPP TS 24.008. They by the way also specified the same mechanism for IPv4, so there's now two competing methods on how to provision IPv4 DNS server information: IPCP and the new method.

In any case, after some more hacking, OpenGGSN can now also provide DNS server information to the MS/UE. And once that was implemented, I had actual live uesr IPv6 data over a full Osmocom cellular stack!

Summary

We now have working IPv6 User IP in OpenGGSN. Together with the rest of the Osmocom stack you can operate a private GPRS, EGPRS, UMTS or HSPA network that provide end-to-end transparent, routed IPv6 connectivity to mobile devices.

All in all, it took much longer than nneeded, and the following questions remain in my mind:

why did the IETF not specify IP6CP capabilities to configure DNS servers?
why the complex two-stage address configuration with PDP EUA allocation for the link-local address first and then stateless autoconfiguration?
why don't we simply allocate the entire prefix via the End User Address information element on the signaling plane? For sure next to the 16byte address we could have put one byte for prefix-length?
why do I see duplication detection flavour neighbour solicitations from Qualcomm based phones on what is a point-to-point link with exactly two devices: The UE and the GGSN?
why do I see link-layer source address options inside the ICMPv6 neighbor and router solicitation from mobile phones, when that option is specifically not to be used on point-to-point links?
why is the smallest prefix that can be allocated a /64? That's such a waste for a point-to-point link with a single device on the other end, and in times of billions of connected IoT devices it will just encourage the use of non-public IPv6 space (i.e. SNAT/MASQUERADING) while wasting large parts of the address space

Some of those choices would have made sense if one would have made it fully compatible with normal IPv6 like e.g. on Ethernet. But implementing ICMPv6 router and neighbor solicitation without getting any benefit such as ability to have multiple prefixes, prefixes of different lengths, I just don't understand why anyone ever thought You can find the code at http://git.osmocom.org/openggsn/log/?h=laforge/ipv6 and the related ticket at https://osmocom.org/issues/2418

Virtual Um interface between OsmoBTS and OsmocomBB

Harald Welte — Tue, 18 Jul 2017 16:00:00 GMT

During the last couple of days, I've been working on completing, cleaning up and merging a Virtual Um interface (i.e. virtual radio layer) between OsmoBTS and OsmocomBB. After I started with the implementation and left it in an early stage in January 2016, Sebastian Stumpf has been completing it around early 2017, with now some subsequent fixes and improvements by me. The combined result allows us to run a complete GSM network with 1-N BTSs and 1-M MSs without any actual radio hardware, which is of course excellent for all kinds of testing scenarios.

The Virtual Um layer is based on sending L2 frames (blocks) encapsulated via GSMTAP UDP multicast packets. There are two separate multicast groups, one for uplink and one for downlink. The multicast nature simulates the shared medium and enables any simulated phone to receive the signal from multiple BTSs via the downlink multicast group.

In OsmoBTS, this is implemented via the new osmo-bts-virtual BTS model.

In OsmocomBB, this is realized by adding virtphy virtual L1, which speaks the same L1CTL protocol that is used between the real OsmcoomBB Layer1 and the Layer2/3 programs such as mobile and the like.

Now many people would argue that GSM without the radio and actual handsets is no fun. I tend to agree, as I'm a hardware person at heart and I am not a big fan of simulation.

Nevertheless, this forms the basis of all kinds of possibilities for automatized (regression) testing in a way and for layers/interfaces that osmo-gsm-tester cannot cover as it uses a black-box proprietary mobile phone (modem). It is also pretty useful if you're traveling a lot and don't want to carry around a BTS and phones all the time, or get some development done in airplanes or other places where operating a radio transmitter is not really a (viable) option.

If you're curious and want to give it a shot, I've put together some setup instructions at the Virtual Um page of the Osmocom Wiki.

How the Osmocom GSM stack is funded

Harald Welte — Thu, 15 Jun 2017 16:00:00 GMT

As the topic has been raised on twitter, I thought I might share a bit of insight into the funding of the Osmocom Cellular Infrastructure Projects.

Keep in mind: Osmocom is a much larger umbrella project, and beyond the Networks-side cellular stack is home many different community-based projects around open source mobile communications. All of those have started more or less as just for fun projects, nothing serious, just a hobby [1]

The projects implementing the network-side protocol stacks and network elements of GSM/GPRS/EGPRS/UMTS cellular networks are somewhat the exception to that, as they have evolved to some extent professionalized. We call those projects collectively the Cellular Infrastructure projects inside Osmocom. This post is about that part of Osmocom only

History

From late 2008 through 2009, People like Holger and I were working on bs11-abis and later OpenBSC only in our spare time. The name Osmocom didn't even exist back then. There was a strong technical community with contributions from Sylvain Munaut, Andreas Eversberg, Daniel Willmann, Jan Luebbe and a few others. None of this would have been possible if it wasn't for all the help we got from Dieter Spaar with the BS-11 [2]. We all had our dayjob in other places, and OpenBSC work was really just a hobby. People were working on it, because it was where no FOSS hacker has gone before. It was cool. It was a big and pleasant challenge to enter the closed telecom space as pure autodidacts.

Holger and I were doing freelance contract development work on Open Source projects for many years before. I was mostly doing Linux related contracting, while Holger has been active in all kinds of areas throughout the FOSS software stack.

In 2010, Holger and I saw some first interest by companies into OpenBSC, including Netzing AG and On-Waves ehf. So we were able to spend at least some of our paid time on OpenBSC/Osmocom related contract work, and were thus able to do less other work. We also continued to spend tons of spare time in bringing Osmocom forward. Also, the amount of contract work we did was only a fraction of the many more hours of spare time.

In 2011, Holger and I decided to start the company sysmocom in order to generate more funding for the Osmocom GSM projects by means of financing software development by product sales. So rather than doing freelance work for companies who bought their BTS hardware from other places (and spent huge amounts of cash on that), we decided that we wanted to be a full solution supplier, who can offer a complete product based on all hardware and software required to run small GSM networks.

The only problem is: We still needed an actual BTS for that. Through some reverse engineering of existing products we figured out who one of the ODM suppliers for the hardware + PHY layer was, and decided to develop the OsmoBTS software to do so. We inherited some of the early code from work done by Andreas Eversberg on the jolly/bts branch of OsmocomBB (thanks), but much was missing at the time.

What follows was Holger and me working several years for free [3], without any salary, in order to complete the OsmoBTS software, build an embedded Linux distribution around it based on OE/poky, write documentation, etc. and complete the first sysmocom product: The sysmoBTS 1002

We did that not because we want to get rich, or because we want to run a business. We did it simply because we saw an opportunity to generate funding for the Osmocom projects and make them more sustainable and successful. And because we believe there is a big, gaping, huge vacuum in terms of absence of FOSS in the cellular telecom sphere.

Funding by means of sysmocom product sales

Once we started to sell the sysmoBTS products, we were able to fund Osmocom related development from the profits made on hardware / full-system product sales. Every single unit sold made a big contribution towards funding both the maintenance as well as the ongoing development on new features.

This source of funding continues to be an important factor today.

Funding by means of R&D contracts

The probably best and most welcome method of funding Osmocom related work is by means of R&D projects in which a customer funds our work to extend the Osmocom GSM stack in one particular area where he has a particular need that the existing code cannot fulfill yet.

This kind of project is the ideal match, as it shows where the true strength of FOSS is: Each of those customers did not have to fund the development of a GSM stack from scratch. Rather, they only had to fund those bits that were missing for their particular application.

Our reference for this is and has been On-Waves, who have been funding development of their required features (and bug fixing etc.) since 2010.

We've of course had many other projects from a variety of customers over over the years. Last, but not least, we had a customer who willingly co-funded (together with funds from NLnet foundation and lots of unpaid effort by sysmocom) the 3G/3.5G support in the Osmocom stack.

The problem here is:

we have not been able to secure anywhere nearly as many of those R&D projects within the cellular industry, despite believing we have a very good foundation upon which we can built. I've been writing many exciting technical project proposals
you almost exclusively get funding only for new features. But it's very hard to get funding for the core maintenance work. The bug-fixing, code review, code refactoring, testing, etc.

So as a result, the profit margin you have on selling R&D projects is basically used to (do a bad job of) fund those bits and pieces that nobody wants to pay for.

Funding by means of customer support

There is a way to generate funding for development by providing support services. We've had some success with this, but primarily alongside the actual hardware/system sales - not so much in terms of pure software-only support.

Also, providing support services from a R&D company means:

either you distract your developers by handling support inquiries. This means they will have less time to work on actual code, and likely get side tracked by too many issues that make it hard to focus
or you have to hire separate support staff. This of course means that the size of the support business has to be sufficiently large to not only cover the cots of hiring + training support staff, but also still generate funding for the actual software R&D.

We've tried shortly with the second option, but fallen back to the first for now. There's simply not sufficient user/admin type support business to rectify dedicated staff for that.

Funding by means of cross-subsizing from other business areas

sysmocom also started to do some non-Osmocom projects in order to generate revenue that we can feed again into Osmocom projects. I'm not at liberty to discuss them in detail, but basically we've been doing pretty much anything from

custom embedded Linux board designs
M2M devices with GSM modems
consulting gigs
public tendered research projects

Profits from all those areas went again into Osmocom development.

Last, but not least, we also operate the sysmocom webshop. The profit we make on those products also is again immediately re-invested into Osmocom development.

Funding by grants

We've had some success in securing funding from NLnet Foundation for specific features. While this is useful, the size of their projects grants of up to EUR 30k is not a good fit for the scale of the tasks we have at hand inside Osmocom. You may think that's a considerable amount of money? Well, that translates to 2-3 man-months of work at a bare cost-covering rate. At a team size of 6 developers, you would theoretically have churned through that in two weeks. Also, their focus is (understandably) on Internet and IT security, and not so much cellular communications.

There are of course other options for grants, such as government research grants and the like. However, they require long-term planning, they require you to match (i.e. pay yourself) a significant portion, and basically mandate that you hire one extra person for doing all the required paperwork and reporting. So all in all, not a particularly attractive option for a very small company consisting of die hard engineers.

Funding by more BTS ports

At sysmocom, we've been doing some ports of the OsmoBTS + OsmoPCU software to other hardware, and supporting those other BTS vendors with porting, R&D and support services.

If sysmocom was a classic BTS vendor, we would not help our "competition". However, we are not. sysmocom exists to help Osmocom, and we strongly believe in open systems and architectures, without a single point of failure, a single supplier for any component or any type of vendor lock-in.

So we happily help third parties to get Osmocom running on their hardware, either with a proprietary PHY or with OsmoTRX.

However, we expect that those BTS vendors also understand their responsibility to share the development and maintenance effort of the stack. Preferably by dedicating some of their own staff to work in the Osmocom community. Alternatively, sysmocom can perform that work as paid service. But that's a double-edged sword: We don't want to be a single point of failure.

Osmocom funding outside of sysmocom

Osmocom is of course more than sysmocom. Even for the cellular infrastructure projects inside Osmocom is true: They are true, community-based, open, collaborative development projects. Anyone can contribute.

Over the years, there have been code contributions by e.g. Fairwaves. They, too, build GSM base station hardware and use that as a means to not only recover the R&D on the hardware, but also to contribute to Osmocom. At some point a few years ago, there was a lot of work from them in the area of OsmoTRX, OsmoBTS and OsmoPCU. Unfortunately, in more recent years, they have not been able to keep up the level of contributions.

There are other companies engaged in activities with and around Osmcoom. There's Rhizomatica, an NGO helping indigenous communities to run their own cellular networks. They have been funding some of our efforts, but being an NGO helping rural regions in developing countries, they of course also don't have the deep pockets. Ideally, we'd want to be the ones contributing to them, not the other way around.

State of funding

We're making some progress in securing funding from players we cannot name [4] during recent years. We're also making occasional progress in convincing BTS suppliers to chip in their share. Unfortunately there are more who don't live up to their responsibility than those who do. I might start calling them out by name one day. The wider community and the public actually deserves to know who plays by FOSS rules and who doesn't. That's not shaming, it's just stating bare facts.

Which brings us to:

sysmocom is in an office that's actually too small for the team, equipment and stock. But we certainly cannot afford more space.
we cannot pay our employees what they could earn working at similar positions in other companies. So working at sysmocom requires dedication to the cause :)
Holger and I have invested way more time than we have ever paid us, even more so considering the opportunity cost of what we would have earned if we'd continued our freelance Open Source hacker path
we're [just barely] managing to pay for 6 developers dedicated to Osmocom development on our payroll based on the various funding sources indicated above

Nevertheless, I doubt that any such a small team has ever implemented an end-to-end GSM/GPRS/EGPRS network from RAN to Core at comparative feature set. My deepest respects to everyone involved. The big task now is to make it sustainable.

Summary

So as you can see, there's quite a bit of funding around. However, it always falls short of what's needed to implement all parts properly, and even not quite sufficient to keep maintaining the status quo in a proper and tested way. That can often be frustrating (mostly to us but sometimes also to users who run into regressions and oter bugs). There's so much more potential. So many things we wanted to add or clean up for a long time, but too little people interested in joining in, helping out - financially or by writing code.

On thing that is often a challenge when dealing with traditional customers: We are not developing a product and then selling a ready-made product. In fact, in FOSS this would be more or less suicidal: We'd have to invest man-years upfront, but then once it is finished, everyone can use it without having to partake in that investment.

So instead, the FOSS model requires the customers/users to chip in early during the R&D phase, in order to then subsequently harvest the fruits of that.

I think the lack of a FOSS mindset across the cellular / telecom industry is the biggest constraining factor here. I've seen that some 20-15 years ago in the Linux world. Trust me, it takes a lot of dedication to the cause to endure this lack of comprehension so many years later.

Playing back GSM RTP streams, RTP-HR bugs

Harald Welte — Sun, 28 May 2017 16:00:00 GMT

Chapter 0: Problem Statement

In an all-IP GSM network, where we use Abis, A and other interfaces within the cellular network over IP transport, the audio of voice calls is transported inside RTP frames. The codec payload in those RTP frames is the actual codec frame of the respective cellular voice codec. In GSM, there are four relevant codecs: FR, HR, EFR and AMR.

Every so often during the (meanwhile many years of ) development of Osmocom cellular infrastructure software it would have been useful to be able to quickly play back the audio for analysis of given issues.

However, until now we didn't have that capability. The reason is relatively simple: In Osmocom, we genally don't do transcoding but simply pass the voice codec frames from left to right. They're only transcoded inside the phones or inside some external media gateway (in case of larger networks).

Chapter 1: GSM Audio Pocket Knife

Back in 2010, when we were very actively working on OsmocomBB, the telephone-side GSM protocol stack implementation, Sylvain Munaut wrote the GSM Audio Pocket Knife (gapk) in order to be able to convert between different formats (representations) of codec frames. In cellular communcations, everyoe is coming up with their own representation for the codec frames: The way they look on E1 as a TRAU frame is completely different from how RTP payload looks like, or what the TI Calypso DSP uses internally, or what a GSM Tester like the Racal 61x3 uses. The differences are mostly about data types used, bit-endinanness as well as padding and headers. And of course those different formats exist for each of the four codecs :/

In 2013 I first added simplistic RTP support for FR-GSM to gapk, which was sufficient for my debugging needs back then. Still, you had to save the decoded PCM output to a file and play that back, or use a pipe into aplay.

Last week, I picked up this subject again and added a long series of patches to gapk:

support for variable-length codec frames (required for AMR support)
support for AMR codec encode/decode using libopencore-amrnb
support of all known RTP payload formats for all four codecs
support for direct live playback to a sound card via ALSA

All of the above can now be combined to make GAPK bind to a specified UDP port and play back the RTP codec frames that anyone sends to that port using a command like this:

$ gapk -I 0.0.0.0/30000 -f rtp-amr -A default -g rawpcm-s16le

I've also merged a chance to OsmoBSC/OsmoNITB which allows the administrator to re-direct the voice of any active voice channel towards a user-specified IP address and port. Using that you can simply disconnect the voice stream from its normal destination and play back the audio via your sound card.

Chapter 2: Bugs in OsmoBTS GSM-HR

While going through the exercise of implementing the above extension to gapk, I had lots of trouble to get it to work for GSM-HR.

After some more digging, it seems there are two conflicting specification on how to format the RTP payload for half-rate GSM:

ETSI TS 101 318 from 1998
IETF RFC 5993 from 2010

In Osmocom, we claim to implement RFC5993, but it turned out that (at least) osmo-bts-sysmo (for sysmoBTS) was actually implementing the ETSI format instead.

And even worse, osmo-bts-sysmo gets event the ETSI format wrong. Each of the codec parameters (which are unaligned bit-fields) are in the wrong bit-endianness :(

Both the above were coincidentially also discovered by Sylvain Munaut during operating of the 32C3 GSM network in December 2015 and resulted the two following "work around" patches: * HACK for HR * HACK: Fix the bit order in HR frames

Those merely worked around those issues in the rtp_proxy of OsmoNITB, rather than addressing the real issue. That's ok, they were "quick" hacks to get something working at all during a four-day conference. I'm now working on "real" fixes in osmo-bts-sysmo. The devil is of course in the details, when people upgrade one BTS but not the other and want to inter-operate, ...

It yet remains to be investigated how osmo-bts-trx and other osmo-bts ports behave in this regard.

Chapter 3: Conclusions

Most definitely it is once again a very clear sign that more testing is required. It's tricky to see even wih osmo-gsm-tester, as GSM-HR works between two phones or even two instances of osmo-bts-sysmo, as both sides of the implementation have the same (wrong) understanding of the spec.

Given that we can only catch this kind of bug together with the hardware (the DSP runs the PHY code), pure unit tests wouldn't catch it. And the end-to-end test is also not very well suited to it. It seems to call for something in betewen. Something like an A-bis interface level test.

We need more (automatic) testing. I cannot say that often enough. The big challenge is how to convince contributors and customers that they should invest their time and money there, rather than yet-another (not automatically tested) feature?

OsmoCon 2017 Updates: Travel Grants and Schedule

Harald Welte — Sun, 26 Mar 2017 16:00:00 GMT

April 21st is approaching fast, so here some updates. I'm particularly happy that we now have travel grants available. So if the travel expenses were preventing you from attending so far: This excuse is no longer valid!

Get your ticket now, before it is too late. There's a limited number of seats available.

OsmoCon 2017 Schedule

The list of talks for OsmoCon 2017 has been available for quite some weeks, but today we finally published the first actual schedule.

As you can see, the day is fully packed with talks about Osmocom cellular infrastructure projects. We had to cut some talk slots short (30min instead of 45min), but I'm confident that it is good to cover a wider range of topics, while at the same time avoiding fragmenting the audience with multiple tracks.

OsmoCon 2017 Travel Grants

We are happy to announce that we have received donations to permit for providing travel grants!

This means that any attendee who is otherwise not able to cover their travel to OsmoCon 2017 (e.g. because their interest in Osmocom is not related to their work, or because their employer doesn't pay the travel expenses) can now apply for such a travel grant.

For more details see OsmoCon 2017 Travel Grants and/or contact osmocon2017@sysmocom.de.

Osmocom Conference 2017 on April 21st

Harald Welte — Tue, 31 Jan 2017 16:00:00 GMT

I'm very happy that in 2017, we will have the first ever technical conference on the Osmocom cellular infrastructure projects.

For many years, we have had a small, invitation only event by Osmocom developers for Osmocom developers called OsmoDevCon. This was fine for the early years of Osmocom, but during the last few years it became apparent that we also need a public event for our many users. Those range from commercial cellular operators to community based efforts like Rhizomatica, and of course include the many research/lab type users with whom we started.

So now we'll have the public OsmoCon on April 21st, back-to-back with the invitation-only OsmoDevcon from April 22nd through 23rd.

I'm hoping we can bring together a representative sample of our user base at OsmoCon 2017 in April. Looking forward to meet you all. I hope you're also curious to hear more from other users, and of course the development team.

Regards,: Harald

Developers wanted for Osmocom GSM related work

Harald Welte — Sun, 01 May 2016 16:00:00 GMT

Right now I'm feeling sad. I really shouldn't, but I still do.

Many years ago I started OpenBSC and Osmocom in order to bring Free Software into an area where it barely existed before: Cellular Infrastructure. For the first few years, it was "just for fun", without any professional users. A FOSS project by enthusiasts. Then we got some commercial / professional users, and with them funding, paying for e.g. Holger and my freelance work. Still, implementing all protocol stacks, interfaces and functional elements of GSM and GPRS from the radio network to the core network is something that large corporations typically spend hundreds of man-years on. So funding for Osmocom GSM implementations was always short, and we always tried to make the best out of it.

After Holger and I started sysmocom in 2011, we had a chance to use funds from BTS sales to hire more developers, and we were growing our team of developers. We finally could pay some developers other than ourselves from working on Free Software cellular network infrastructure.

In 2014 and 2015, sysmocom got side-tracked with some projects where Osmocom and the cellular network was only one small part of a much larger scope. In Q4/2015 and in 2016, we are back on track with focussing 100% at Osmocom projects, which you can probably see by a lot more associated commits to the respective project repositories.

By now, we are in the lucky situation that the work we've done in the Osmocom project on providing Free Software implementations of cellular technologies like GSM, GPRS, EDGE and now also UMTS is receiving a lot of attention. This attention translates into companies approaching us (particularly at sysmocom) regarding funding for implementing new features, fixing existing bugs and short-comings, etc. As part of that, we can even work on much needed infrastructural changes in the software.

So now we are in the opposite situation: There's a lot of interest in funding Osmocom work, but there are few people in the Osmocom community interested and/or capable to follow-up to that. Some of the early contributors have moved into other areas, and are now working on proprietary cellular stacks at large multi-national corporations. Some others think of GSM as a fun hobby and want to keep it that way.

At sysmocom, we are trying hard to do what we can to keep up with the demand. We've been looking to add people to our staff, but right now we are struggling only to compensate for the regular fluctuation of employees (i.e. keep the team size as is), let alone actually adding new members to our team to help to move free software cellular networks ahead.

I am struggling to understand why that is. I think Free Software in cellular communications is one of the most interesting and challenging frontiers for Free Software to work on. And there are many FOSS developers who love nothing more than to conquer new areas of technology.

At sysmocom, we can now offer what would have been my personal dream job for many years:

paid work on Free Software that is available to the general public, rather than something only of value to the employer
interesting technical challenges in an area of technology where you will not find the answer to all your problems on stackoverflow or the like
work in a small company consisting almost entirely only of die-hard engineers, without corporate managers, marketing departments, etc.
work in an environment free of Microsoft and Apple software or cloud services; use exclusively Free Software to get your work done

I would hope that more developers would appreciate such an environment. If you're interested in helping FOSS cellular networks ahead, feel free to have a look at http://sysmocom.de/jobs or contact us at jobs@sysmocom.de. Together, we can try to move Free Software for mobile communications to the next level!

You can now install a GSM network using apt-get

Harald Welte — Sun, 27 Mar 2016 16:00:00 GMT

This is great news: You can now install a GSM network using apt-get!

Thanks to the efforts of Debian developer Ruben Undheim, there's now an OpenBSC (with all its flavors like OsmoBSC, OsmoNITB, OsmoSGSN, ...) package in the official Debian repository.

Here is the link to the e-mail indicating acceptance into Debian: https://tracker.debian.org/news/755641

I think for the past many years into the OpenBSC (and wider Osmocom) projects I always assumed that distribution packaging is not really something all that important, as all the people using OpenBSC surely would be technical enough to build it from the source. And in fact, I believe that building from source brings you one step closer to actually modifying the code, and thus contribution.

Nevertheless, the project has matured to a point where it is not used only by developers anymore, and particularly also (god beware) by people with limited experience with Linux in general. That such people still exist is surprisingly hard to realize for somebody like myself who has spent more than 20 years in Linux land by now.

So all in all, today I think that having packages in a Distribution like Debian actually is important for the further adoption of the project - pretty much like I believe that more and better public documentation is.

Looking forward to seeing the first bug reports reported through bugs.debian.org rather than https://projects.osmocom.org/ . Once that happens, we know that people are actually using the official Debian packages.

As an unrelated side note, the Osmocom project now also has nightly builds available for Debian 7.0, Debian 8.0 and Ubunut 14.04 on both i586 and x86_64 architecture from https://build.opensuse.org/project/show/network:osmocom:nightly. The nightly builds are for people who want to stay on the bleeding edge of the code, but who don't want to go through building everything from scratch. See Holgers post on the openbsc mailing list for more information.

Osmocom.org migrating to redmine

Harald Welte — Sat, 20 Feb 2016 16:00:00 GMT

In 2008, we started bs11-abis, which was shortly after renamed to OpenBSC. At the time it seemed like a good idea to use trac as the project management system, to have a wiki and an issue tracker.

When further Osmocom projects like OsmocomBB, OsmocomTETRA etc. came around, we simply replicated that infrastructure: Another trac instance with the same theme, and a shared password file.

The problem with this (and possibly the way we used it) is:

it doesn't scale, as creating projects is manual, requires a sysadmin and is time-consuming. This meant e.g. SIMtrace was just a wiki page in the OsmocomBB trac installation + associated http redirect, causing some confusion.
issues can not easily be moved from one project to another, or have cross-project relationships (like, depend on an issue in another project)
we had to use an external planet in order to aggregate the blog of each of the trac instances
user account management the way we did it required shell access to the machine, meaning user account applications got dropped due to the effort involved. My apologies for that.

Especially the lack of being able to move pages and tickets between trac's has resulted in a suboptimal use of the tools. If we first write code as part of OpenBSC and then move it to libosmocore, the associated issues + wiki pages should be moved to a new project.

At the same time, for the last 5 years we've been successfully using redmine inside sysmocom to keep track of many dozens of internal projects.

So now, finally, we (zecke, tnt, myself) have taken up the task to migrate the osmocom.org projects into redmine. You can see the current status at http://projects.osmocom.org/. We could create a more comprehensive project hierarchy, and give libosmocore, SIMtrace, OsmoSGSN and many others their own project.

Thanks to zecke for taking care of the installation/sysadmin part and the initial conversion!

Unfortunately the conversion from trac to redmine wiki syntax (and structure) was not as automatic and straight-forward as one would have hoped. But after spending one entire day going through the most important wiki pages, things are looking much better now. As a side effect, I have had a more comprehensive look into the history of all of our projects than ever before :)

Still, a lot of clean-up and improvement is needed until I'm happy, particularly splitting the OpenBSC wiki into separate OsmoBSC, OsmoNITB, OsmoBTS, OsmoPCU and OsmoSGSN wiki's is probably still going to take some time.

If you would like to help out, feel free to register an account on projects.osmocom.org (if you don't already have one from the old trac projects) and mail me for write access to the project(s) of your choice.

Possible tasks include

putting pages into a more hierarchic structure (there's a parent/child relationship in redmine wikis)
fixing broken links due to page renames / wiki page moves
creating a new redmine 'Project' for your favorite tool that has a git repo on http://git.osmocom.org/ and writing some (at least initial) documentation about it.

You don't need to be a software developer for that!

32C3 is over, GSM and GPRS was running fine, osmo-iuh progress

Harald Welte — Wed, 30 Dec 2015 16:00:00 GMT

The 32C3 GSM Network

32C3 was great from the Osmocom perspective: We could again run our own cellular network at the event in order to perform load testing with real users. We had 7 BTSs running, each with a single TRX. What was new compared to previous years:

OsmoPCU is significantly more robust and stable due to the efforts of Jacob Erlbeck at sysmocom. This means that GPRS is now actually still usable in severe overload situations, like 1000 subscribers sharing only very few kilobits. Of course it will be slow, but at least data still passes through as much as that's possible.
We were using half-rate traffic channels from day 2 onwards, in order to enhance capacity. Phones supporting AMR-HR would use that, but then there are lots of old phones that only do classic HR (v1). OsmoNITB with internal MNCC handler supports TCH/H with HR and AMR for at least five years, but the particular combination of OsmoBTS + OsmoNITB + lcr (all master branches) was not yet deployed at previous CCC event networks so far.

Being forced to provide classic HR codec actually revealed several bugs in the existing code:

OsmoBTS (at least with the sysmoBTS hardware) is using bit ordering that is not compliant to what the spec says on how GSM-HR frames should be put into RTP frames. We didn't realize this so far, as handing frames from one sysmoBTS to another sysmoBTS of course works, as both use the same (wrong) bit ordering.
The ETSI reference implementation of the HR codec has lots of global/static variables, and thus doesn't really support running multiple transcoders in parallel. This is however what lcr was trying (and needing) to do, and it of course failed as state from one transcoder instance was leaking into another. The problem is simple, but the solution not so simple. If you want to avoid re-structuring the entire code in very intrusive ways or running one thread per transcoder instance, then the only solution was to basically memcpy() the entire data section of the transcoding library every time you switch the state from one transcoder instance to the other. It's surprisingly difficult to learn the start + size of that data section at runtime in a portable way, though.

Thanks to our resident voice codec expert Sylvain for debugging and fixing the above two problems.

Thanks also to Daniel and Ulli for taking care of the actual logistics of bringing + installing (+ later unmounting) all associated equipment.

Thanks furthermore to Kevin who has been patiently handling the 'Level 2 Support' cases of people with various problems ending up in the GSM room.

It's great that there is a team taking care of those real-world test networks. We learn a lot more about our software under heavy load situations this way.

osmo-iuh progress + talk

I've been focussing basically full day (and night) over the week ahead of Christmas and during Christmas to bring the osmo-iuh code into a state where we could do a end-to-end demo with a regular phone + hNodeB + osmo-hnbgw + osmo-sgsn + openggsn. Unfortunately I only got it up to the point where we do the PDP CONTEXT ACTIVATION on the signalling plane, with no actual user data going back and forth. And then, for strange reasons, I couldn't even demo that at the end of the talk. Well, in either case, the code has made much progress.

The video of the talk can be found at https://media.ccc.de/v/32c3-7412-running_your_own_3g_3_5g_network#video

meeting friends

The annual CCC congress is always an event where you meet old friends and colleagues. It was great talking to Stefan, Dimitri, Kevin, Nico, Sylvain, Jochen, Sec, Schneider, bunnie and many other hackers. After the event is over, I wish I could continue working together with all those folks the rest of the year, too :/

Some people have been missed dearly. Absence from the CCC congress is not acceptable. You know who you are, if you're reading this ;)

Anyone interested in supporting SMPP interworking at 32C3?

Harald Welte — Sat, 05 Dec 2015 16:00:00 GMT

Sylvain brought this up yesterday: Wouldn't it be nice to have some degree of SMS interfacing from OpenBSC/OsmoNITB to the real world at 32C3? It is something that we've never tried so far, and thus definitely worthy of testing.

Of course, full interworking is not possible without assigning public MSISDN to all internal subscribers / 'extensions' how we call them.

But what would most certainly work is to have at least outbound SMS working by means of an external SMPP interface.

The OsmoNITB-internal SMSC speaks SMPP already (in the SMSC role), so we would need to implement some small amount of glue logic that behaves as ESME (external SMS entity) towards both OsmoNITB as well as some public SMS operator/reseller that speaks SMPP again.

Now of course, sending SMS to public operators doesn't come for free. So in case anyone reading this has access to SMPP at public operators, resellers, SMS hubs, it would be interesting to see if there is a chance for some funding/sponsoring of that experiment.

Feel free to contact me if you see a way to make this happen.

Python tool to talk to OsmoNITB MNCC interface

Harald Welte — Tue, 01 Dec 2015 16:00:00 GMT

I've been working on a small python tool that can be used to attach to the MNCC interface of OsmoNITB. It implements the 04.08 CC state machine with our MNCC primitives, including support for RTP bridge mode of the voice streams.

The immediate first use case for this was to be able to generate MT calls to a set of known MSISDNs and load all 14 TCH/H channels of a single-TRX BTS. It will connect the MT calls in pairs, so you end up with 7 MS-to-MS calls.

The first working version of the tool is available from

http://git.osmocom.org/mncc-python/
git clone git://git.osmocom.org/mncc-python

The code is pretty hacky in some places. That's partially due to the fact that I'm much more familiar in the C, Perl and Erlang world than in python. Still I thought it's a good idea to do it in python to enable more people to use/edit/contribute to it.

I'm happy for review / cleanup suggestion by people with more Python-foo than I have.

Architecturally, I decided to do things a bit erlang-like, where we have finite state machines in an actor models, and message passing between the actors. This is what happens with the GsmCallFsm()'s, which are created by the GsmCallConnector() representing both legs of a call and the MnccActor() that wraps the MNCC socket towards OsmoNITB.

The actual encoding/decoding of MNCC messages is auto-generated from the mncc header file #defines, enums and c-structures by means of ctypes code generation.

mncc_test.py currently drops you into a python shell where you can e.g. start more / new calls by calling functions like connect_call("7839", "3802") from that shell. Exiting the shell by quit() or Ctrl+C will terminate all call FSMs and terminate.

LaForge's home page (Posts about openbsc)

Osmocom jenkins test suite execution

Automatic Testing in Osmocom

OsmoSTP (M3UA/SUA)

OpenGGSN (GTP)

Further Work

IPv6 User Plane support in Osmocom

Preface

OpenGGSN IPv6 PDP Context Support

Summary

Virtual Um interface between OsmoBTS and OsmocomBB

How the Osmocom GSM stack is funded

History

Funding by means of sysmocom product sales

Funding by means of R&D contracts

Funding by means of customer support

Funding by means of cross-subsizing from other business areas

Funding by grants

Funding by more BTS ports

Osmocom funding outside of sysmocom

State of funding

Summary

Playing back GSM RTP streams, RTP-HR bugs

Chapter 0: Problem Statement

Chapter 1: GSM Audio Pocket Knife

Chapter 2: Bugs in OsmoBTS GSM-HR

Chapter 3: Conclusions

OsmoCon 2017 Updates: Travel Grants and Schedule

OsmoCon 2017 Schedule

OsmoCon 2017 Travel Grants

OsmoCon 2017 Social Event

Osmocom Conference 2017 on April 21st

Developers wanted for Osmocom GSM related work

You can now install a GSM network using apt-get

Osmocom.org migrating to redmine

32C3 is over, GSM and GPRS was running fine, osmo-iuh progress

The 32C3 GSM Network

osmo-iuh progress + talk

meeting friends

Anyone interested in supporting SMPP interworking at 32C3?

Python tool to talk to OsmoNITB MNCC interface