Black Hole entropy and apparent horizon

The black hole entropy is given by $S_{BH}= \frac{k_BA}{4l_{pl}^2}\,,~~~~~~(1)$ where $A$ is the area of the black hole. As far as I know, this entropy is constant for any observer, and it's an intrinsic property of the BH. This is precisely where I get lost. In an isolated system where there are only me and the black hole, I can locally choose to be in a free-falling frame of reference (so the information about the Schwarzschild radius in the metric has "disappeared"). Then, when I take a look at the BH, the only thing I can measure in my frame is the apparent horizon, which may not be the same as the actual horizon of the black hole. I also get the feeling that, in my frame of reference, I have no way to tell if I'm far from a big BH or if I'm close to a small one. Therefore, I don't understand why every observer should agree on the entropy given in (1).