The rotary encoder works. I used interrupts on pin state change to track movements. This works great. In previous projects I used polling. This caused problems for fast rotations.

It is possible to make the sweep (Sawtooth) using the PWM output. Just increase the duty cycle starting at 0% until the maximum is reached. Then filter the signal using a low pass filter. I have used a simple low pass RC filter at the PWM output. The signal looks good for slow sweep frequencies. For higher frequencies the ramp is not a straight line. I think it can be solved by using a higher order filter.

I have added a switch to select the function of the rotary encoder. Two modes are foreseen. Center frequency and span (MHz/division). In span mode the sweep frequency has to be adapted. Next step is to make this possible. I will try to clean up the ramp signal as well.
I bought some switches (4013) and opamps (TL072) to make a voltage divider for the sweep signal.

I have just ordered DA converters. They will be used to set the center frequency. I hope they will arrive soon.