Low-power wide-area networks (LPWAN) are becoming the wireless backbone for modern business processes and municipal administration. LoRaWAN, which stands for long-range wide-area network, is a recent medium access control (MAC) layer protocol competing for this market. It stands out by its open operator model and a novel modulation technique. With LoRaWAN and other communication technologies are becoming a dependency for more and more aspects of today's society, the question for their security and reliability comes up. Previous researches on the topic have already revealed vulnerabilities in the first LoRaWAN specification, which have been partly mitigated in the most recent LoRaWAN 1.1. However, related studies often provide only theoretical results or consider practical scenarios only on a specific, small scale. In this thesis, we present a LoRaWAN security evaluation framework that allows field-testing the security and reliability characteristics of actual LoRaWAN deployments. This provides not only reproducible results but also allows making a comparison between defined versions of the specification and LoRaWAN software. Before expounding implementation details, we provide a literature survey on LoRaWAN vulnerabilities and attacks to identify interesting aspects for further evaluation. From our experimental results, we show that jamming is a serious threat to the availability of LoRaWAN networks. Furthermore, we demonstrate the practical applicability of two replay attacks against a selection of LoRaWAN software and illustrate why they will remain relevant for years due to backward compatibility.